Sep 06 19:11:04.507302 ip-10-0-129-252 systemd[1]: Starting Kubernetes Kubelet...
Sep 06 19:11:04.956335 ip-10-0-129-252 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 06 19:11:04.956335 ip-10-0-129-252 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 06 19:11:04.956335 ip-10-0-129-252 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 06 19:11:04.956335 ip-10-0-129-252 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 06 19:11:04.956335 ip-10-0-129-252 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 06 19:11:04.959195 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.959112    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 06 19:11:04.964314 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964297    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 19:11:04.964314 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964314    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 19:11:04.964385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964318    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 19:11:04.964385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964321    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 19:11:04.964385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964324    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 19:11:04.964385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964327    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 19:11:04.964385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964330    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 19:11:04.964385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964333    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 19:11:04.964385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964336    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 19:11:04.964385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964339    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 19:11:04.964385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964342    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 19:11:04.964385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964345    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 19:11:04.964385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964347    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 19:11:04.964385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964350    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 19:11:04.964385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964353    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 19:11:04.964385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964356    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 19:11:04.964385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964358    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 19:11:04.964385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964361    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 19:11:04.964385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964364    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 19:11:04.964385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964372    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 19:11:04.964385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964374    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 19:11:04.964385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964377    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 19:11:04.964888 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964379    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 19:11:04.964888 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964382    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 19:11:04.964888 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964385    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 19:11:04.964888 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964387    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 19:11:04.964888 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964390    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 19:11:04.964888 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964393    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 19:11:04.964888 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964395    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 19:11:04.964888 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964399    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 19:11:04.964888 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964401    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 19:11:04.964888 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964404    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 19:11:04.964888 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964406    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 19:11:04.964888 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964409    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 19:11:04.964888 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964411    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 19:11:04.964888 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964415    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 19:11:04.964888 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964418    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 19:11:04.964888 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964420    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 19:11:04.964888 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964422    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 19:11:04.964888 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964425    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 19:11:04.964888 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964427    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 19:11:04.964888 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964429    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 19:11:04.965385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964432    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 19:11:04.965385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964434    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 19:11:04.965385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964437    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 19:11:04.965385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964439    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 19:11:04.965385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964441    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 19:11:04.965385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964444    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 19:11:04.965385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964447    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 19:11:04.965385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964449    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 19:11:04.965385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964452    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 19:11:04.965385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964454    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 19:11:04.965385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964457    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 19:11:04.965385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964462    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 19:11:04.965385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964467    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 19:11:04.965385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964470    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 19:11:04.965385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964473    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 19:11:04.965385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964477    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 19:11:04.965385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964480    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 19:11:04.965385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964483    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 19:11:04.965385 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964486    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 19:11:04.965841 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964489    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 19:11:04.965841 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964491    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 19:11:04.965841 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964494    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 19:11:04.965841 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964497    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 19:11:04.965841 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964499    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 19:11:04.965841 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964502    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 19:11:04.965841 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964505    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 19:11:04.965841 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964507    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 19:11:04.965841 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964510    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 19:11:04.965841 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964512    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 19:11:04.965841 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964516    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 19:11:04.965841 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964519    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 19:11:04.965841 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964521    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 19:11:04.965841 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964524    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 19:11:04.965841 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964527    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 19:11:04.965841 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964530    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 19:11:04.965841 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964532    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 19:11:04.965841 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964535    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 19:11:04.965841 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964537    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 19:11:04.966301 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964540    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 19:11:04.966301 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964543    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 19:11:04.966301 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964545    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 19:11:04.966301 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964548    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 19:11:04.966301 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964550    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 19:11:04.966301 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964554    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 19:11:04.966301 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964557    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 19:11:04.966301 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964559    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 19:11:04.966301 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964562    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 19:11:04.966301 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964564    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 19:11:04.966301 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964567    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 19:11:04.966301 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964569    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 19:11:04.966301 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964572    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 19:11:04.966301 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964574    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 19:11:04.966301 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964577    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 19:11:04.966301 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964579    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 19:11:04.966301 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964583    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 19:11:04.966301 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964586    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 19:11:04.966301 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.964589    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 19:11:04.966301 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965274    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 19:11:04.966770 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965281    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 19:11:04.966770 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965283    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 19:11:04.966770 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965287    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 19:11:04.966770 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965290    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 19:11:04.966770 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965293    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 19:11:04.966770 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965296    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 19:11:04.966770 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965299    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 19:11:04.966770 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965301    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 19:11:04.966770 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965305    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 19:11:04.966770 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965309    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 19:11:04.966770 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965312    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 19:11:04.966770 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965315    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 19:11:04.966770 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965317    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 19:11:04.966770 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965320    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 19:11:04.966770 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965322    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 19:11:04.966770 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965325    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 19:11:04.966770 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965327    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 19:11:04.966770 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965330    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 19:11:04.966770 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965334    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 19:11:04.967263 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965337    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 19:11:04.967263 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965340    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 19:11:04.967263 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965343    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 19:11:04.967263 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965345    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 19:11:04.967263 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965348    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 19:11:04.967263 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965352    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 19:11:04.967263 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965354    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 19:11:04.967263 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965357    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 19:11:04.967263 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965361    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 19:11:04.967263 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965365    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 19:11:04.967263 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965368    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 19:11:04.967263 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965370    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 19:11:04.967263 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965373    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 19:11:04.967263 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965376    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 19:11:04.967263 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965379    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 19:11:04.967263 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965381    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 19:11:04.967263 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965384    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 19:11:04.967263 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965387    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 19:11:04.967263 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965389    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 19:11:04.967263 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965392    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 19:11:04.967752 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965395    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 19:11:04.967752 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965397    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 19:11:04.967752 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965400    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 19:11:04.967752 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965403    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 19:11:04.967752 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965405    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 19:11:04.967752 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965409    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 19:11:04.967752 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965411    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 19:11:04.967752 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965414    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 19:11:04.967752 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965416    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 19:11:04.967752 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965419    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 19:11:04.967752 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965422    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 19:11:04.967752 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965424    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 19:11:04.967752 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965428    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 19:11:04.967752 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965430    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 19:11:04.967752 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965433    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 19:11:04.967752 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965435    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 19:11:04.967752 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965438    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 19:11:04.967752 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965440    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 19:11:04.967752 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965443    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 19:11:04.967752 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965445    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 19:11:04.968254 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965448    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 19:11:04.968254 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965450    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 19:11:04.968254 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965452    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 19:11:04.968254 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965455    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 19:11:04.968254 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965457    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 19:11:04.968254 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965460    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 19:11:04.968254 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965462    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 19:11:04.968254 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965464    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 19:11:04.968254 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965467    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 19:11:04.968254 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965470    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 19:11:04.968254 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965472    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 19:11:04.968254 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965474    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 19:11:04.968254 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965478    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 19:11:04.968254 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965480    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 19:11:04.968254 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965483    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 19:11:04.968254 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965485    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 19:11:04.968254 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965488    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 19:11:04.968254 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965491    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 19:11:04.968254 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965493    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 19:11:04.968254 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965495    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 19:11:04.968764 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965498    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 19:11:04.968764 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965500    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 19:11:04.968764 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965503    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 19:11:04.968764 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965505    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 19:11:04.968764 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965508    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 19:11:04.968764 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965514    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 19:11:04.968764 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965517    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 19:11:04.968764 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965521    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 19:11:04.968764 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965524    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 19:11:04.968764 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965527    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 19:11:04.968764 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965530    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 19:11:04.968764 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965532    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 19:11:04.968764 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965535    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 19:11:04.968764 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965538    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 19:11:04.968764 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965540    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 19:11:04.968764 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965543    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 19:11:04.968764 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965545    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 19:11:04.968764 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965547    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 19:11:04.968764 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965550    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 19:11:04.969239 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965885    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 19:11:04.969239 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965890    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 19:11:04.969239 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965893    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 19:11:04.969239 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965896    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 19:11:04.969239 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965899    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 19:11:04.969239 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965901    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 19:11:04.969239 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965904    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 19:11:04.969239 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965906    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 19:11:04.969239 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965909    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 19:11:04.969239 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965915    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 19:11:04.969239 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965917    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 19:11:04.969239 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965936    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 19:11:04.969239 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965939    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 19:11:04.969239 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965941    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 19:11:04.969239 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965944    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 19:11:04.969239 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965947    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 19:11:04.969239 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965949    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 19:11:04.969239 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965952    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 19:11:04.969239 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965955    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 19:11:04.969239 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965957    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 19:11:04.969790 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965960    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 19:11:04.969790 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965963    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 19:11:04.969790 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965965    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 19:11:04.969790 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965967    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 19:11:04.969790 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965971    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 19:11:04.969790 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965974    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 19:11:04.969790 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965976    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 19:11:04.969790 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965979    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 19:11:04.969790 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965981    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 19:11:04.969790 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965984    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 19:11:04.969790 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965986    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 19:11:04.969790 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965989    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 19:11:04.969790 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965991    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 19:11:04.969790 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965994    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 19:11:04.969790 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965996    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 19:11:04.969790 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.965999    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 19:11:04.969790 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966001    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 19:11:04.969790 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966004    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 19:11:04.969790 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966006    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 19:11:04.969790 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966009    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 19:11:04.970288 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966011    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 19:11:04.970288 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966015    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 19:11:04.970288 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966018    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 19:11:04.970288 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966021    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 19:11:04.970288 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966025    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 19:11:04.970288 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966029    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 19:11:04.970288 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966032    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 19:11:04.970288 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966034    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 19:11:04.970288 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966036    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 19:11:04.970288 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966040    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 19:11:04.970288 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966042    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 19:11:04.970288 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966045    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 19:11:04.970288 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966048    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 19:11:04.970288 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966050    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 19:11:04.970288 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966053    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 19:11:04.970288 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966055    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 19:11:04.970288 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966058    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 19:11:04.970288 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966060    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 19:11:04.970288 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966063    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 19:11:04.970288 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966066    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 19:11:04.970774 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966068    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 19:11:04.970774 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966071    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 19:11:04.970774 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966073    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 19:11:04.970774 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966075    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 19:11:04.970774 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966078    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 19:11:04.970774 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966080    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 19:11:04.970774 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966083    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 19:11:04.970774 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966085    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 19:11:04.970774 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966088    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 19:11:04.970774 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966091    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 19:11:04.970774 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966093    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 19:11:04.970774 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966096    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 19:11:04.970774 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966099    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 19:11:04.970774 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966103    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 19:11:04.970774 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966105    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 19:11:04.970774 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966108    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 19:11:04.970774 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966110    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 19:11:04.970774 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966112    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 19:11:04.970774 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966115    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 19:11:04.970774 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966117    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 19:11:04.971303 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966120    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 19:11:04.971303 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966122    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 19:11:04.971303 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966125    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 19:11:04.971303 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966127    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 19:11:04.971303 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966130    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 19:11:04.971303 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966132    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 19:11:04.971303 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966135    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 19:11:04.971303 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966137    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 19:11:04.971303 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966140    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 19:11:04.971303 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966142    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 19:11:04.971303 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966145    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 19:11:04.971303 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966147    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 19:11:04.971303 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966151    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 19:11:04.971303 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966155    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 19:11:04.971303 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966158    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 19:11:04.971303 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966160    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 19:11:04.971303 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966163    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 19:11:04.971303 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966166    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 19:11:04.971303 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966169    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966239    2566 flags.go:64] FLAG: --address="0.0.0.0"
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966274    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966279    2566 flags.go:64] FLAG: --anonymous-auth="true"
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966284    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966288    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966291    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966297    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966301    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966304    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966307    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966310    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966313    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966316    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966319    2566 flags.go:64] FLAG: --cgroup-root=""
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966322    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966325    2566 flags.go:64] FLAG: --client-ca-file=""
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966328    2566 flags.go:64] FLAG: --cloud-provider="external"
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966331    2566 flags.go:64] FLAG: --cluster-dns="[]"
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966336    2566 flags.go:64] FLAG: --cluster-domain=""
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966339    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966342    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966345    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966348    2566 flags.go:64] FLAG: --container-log-max-files="5"
Sep 06 19:11:04.971757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966352    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966355    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966358    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966361    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966363    2566 flags.go:64] FLAG: --contention-profiling="false"
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966366    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966369    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966372    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966375    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966378    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966381    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966384    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966387    2566 flags.go:64] FLAG: --enable-load-reader="false"
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966390    2566 flags.go:64] FLAG: --enable-server="true"
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966393    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966397    2566 flags.go:64] FLAG: --event-burst="100"
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966400    2566 flags.go:64] FLAG: --event-qps="50"
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966403    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966406    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966408    2566 flags.go:64] FLAG: --eviction-hard=""
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966412    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966414    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966417    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966422    2566 flags.go:64] FLAG: --eviction-soft=""
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966424    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 06 19:11:04.972341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966427    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966430    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966433    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966436    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966438    2566 flags.go:64] FLAG: --fail-swap-on="true"
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966441    2566 flags.go:64] FLAG: --feature-gates=""
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966444    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966447    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966450    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966453    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966455    2566 flags.go:64] FLAG: --healthz-port="10248"
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966458    2566 flags.go:64] FLAG: --help="false"
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966461    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-129-252.ec2.internal"
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966464    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966466    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966469    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966472    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966475    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966478    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966480    2566 flags.go:64] FLAG: --image-service-endpoint=""
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966483    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966485    2566 flags.go:64] FLAG: --kube-api-burst="100"
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966488    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966491    2566 flags.go:64] FLAG: --kube-api-qps="50"
Sep 06 19:11:04.972976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966494    2566 flags.go:64] FLAG: --kube-reserved=""
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966497    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966499    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966502    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966505    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966508    2566 flags.go:64] FLAG: --lock-file=""
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966512    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966515    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966518    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966523    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966526    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966529    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966532    2566 flags.go:64] FLAG: --logging-format="text"
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966534    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966538    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966540    2566 flags.go:64] FLAG: --manifest-url=""
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966543    2566 flags.go:64] FLAG: --manifest-url-header=""
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966547    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966550    2566 flags.go:64] FLAG: --max-open-files="1000000"
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966553    2566 flags.go:64] FLAG: --max-pods="110"
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966556    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966559    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966562    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966564    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966567    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 06 19:11:04.973561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966570    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966573    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966579    2566 flags.go:64] FLAG: --node-status-max-images="50"
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966582    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966585    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966588    2566 flags.go:64] FLAG: --pod-cidr=""
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966591    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966596    2566 flags.go:64] FLAG: --pod-manifest-path=""
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966599    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966602    2566 flags.go:64] FLAG: --pods-per-core="0"
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966604    2566 flags.go:64] FLAG: --port="10250"
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966607    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966610    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-007447dd695d1af2e"
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966614    2566 flags.go:64] FLAG: --qos-reserved=""
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966617    2566 flags.go:64] FLAG: --read-only-port="10255"
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966620    2566 flags.go:64] FLAG: --register-node="true"
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966623    2566 flags.go:64] FLAG: --register-with-taints=""
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966626    2566 flags.go:64] FLAG: --registry-burst="10"
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966629    2566 flags.go:64] FLAG: --registry-qps="5"
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966631    2566 flags.go:64] FLAG: --reserved-cpus=""
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966634    2566 flags.go:64] FLAG: --reserved-memory=""
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966638    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966641    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966644    2566 flags.go:64] FLAG: --rotate-certificates="false"
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966646    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 06 19:11:04.974169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966649    2566 flags.go:64] FLAG: --runonce="false"
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966652    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966654    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966657    2566 flags.go:64] FLAG: --seccomp-default="false"
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966660    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966663    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966665    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966668    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966671    2566 flags.go:64] FLAG: --storage-driver-password="root"
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966674    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966676    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966679    2566 flags.go:64] FLAG: --storage-driver-user="root"
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966682    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966685    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966687    2566 flags.go:64] FLAG: --system-cgroups=""
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966690    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966695    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966698    2566 flags.go:64] FLAG: --tls-cert-file=""
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966700    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966704    2566 flags.go:64] FLAG: --tls-min-version=""
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966708    2566 flags.go:64] FLAG: --tls-private-key-file=""
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966711    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966714    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966716    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966719    2566 flags.go:64] FLAG: --v="2"
Sep 06 19:11:04.975268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966723    2566 flags.go:64] FLAG: --version="false"
Sep 06 19:11:04.976291 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966726    2566 flags.go:64] FLAG: --vmodule=""
Sep 06 19:11:04.976291 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966730    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 06 19:11:04.976291 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.966733    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 06 19:11:04.976291 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966840    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 19:11:04.976291 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966845    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 19:11:04.976291 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966848    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 19:11:04.976291 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966851    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 19:11:04.976291 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966854    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 19:11:04.976291 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966856    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 19:11:04.976291 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966859    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 19:11:04.976291 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966862    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 19:11:04.976291 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966864    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 19:11:04.976291 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966866    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 19:11:04.976291 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966869    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 19:11:04.976291 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966871    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 19:11:04.976291 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966873    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 19:11:04.976291 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966876    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 19:11:04.976291 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966879    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 19:11:04.976291 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966882    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 19:11:04.976291 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966884    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 19:11:04.976773 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966887    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 19:11:04.976773 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966890    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 19:11:04.976773 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966892    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 19:11:04.976773 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966895    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 19:11:04.976773 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966897    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 19:11:04.976773 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966901    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 19:11:04.976773 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966906    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 19:11:04.976773 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966909    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 19:11:04.976773 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966911    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 19:11:04.976773 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966914    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 19:11:04.976773 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966917    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 19:11:04.976773 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966934    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 19:11:04.976773 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966937    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 19:11:04.976773 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966940    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 19:11:04.976773 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966943    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 19:11:04.976773 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966946    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 19:11:04.976773 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966949    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 19:11:04.976773 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966952    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 19:11:04.976773 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966954    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 19:11:04.977258 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966957    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 19:11:04.977258 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966959    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 19:11:04.977258 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966962    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 19:11:04.977258 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966965    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 19:11:04.977258 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966967    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 19:11:04.977258 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966970    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 19:11:04.977258 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966973    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 19:11:04.977258 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966975    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 19:11:04.977258 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966978    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 19:11:04.977258 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966980    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 19:11:04.977258 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966985    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 19:11:04.977258 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966987    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 19:11:04.977258 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966989    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 19:11:04.977258 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966992    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 19:11:04.977258 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966995    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 19:11:04.977258 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.966997    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 19:11:04.977258 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967000    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 19:11:04.977258 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967002    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 19:11:04.977258 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967005    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 19:11:04.977258 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967008    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 19:11:04.977737 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967011    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 19:11:04.977737 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967015    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 19:11:04.977737 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967017    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 19:11:04.977737 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967020    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 19:11:04.977737 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967023    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 19:11:04.977737 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967025    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 19:11:04.977737 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967028    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 19:11:04.977737 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967030    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 19:11:04.977737 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967033    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 19:11:04.977737 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967036    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 19:11:04.977737 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967038    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 19:11:04.977737 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967041    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 19:11:04.977737 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967043    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 19:11:04.977737 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967045    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 19:11:04.977737 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967048    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 19:11:04.977737 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967050    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 19:11:04.977737 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967053    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 19:11:04.977737 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967055    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 19:11:04.977737 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967058    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 19:11:04.977737 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967060    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 19:11:04.978233 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967063    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 19:11:04.978233 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967065    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 19:11:04.978233 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967069    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 19:11:04.978233 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967071    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 19:11:04.978233 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967074    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 19:11:04.978233 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967076    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 19:11:04.978233 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967079    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 19:11:04.978233 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967081    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 19:11:04.978233 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967084    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 19:11:04.978233 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967086    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 19:11:04.978233 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967089    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 19:11:04.978233 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967100    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 19:11:04.978233 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967104    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 19:11:04.978233 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967107    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 19:11:04.978233 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967110    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 19:11:04.978233 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967112    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 19:11:04.978233 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967114    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 19:11:04.978233 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967117    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 19:11:04.978233 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967120    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 19:11:04.978233 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967122    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 19:11:04.978723 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967125    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 19:11:04.978723 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967127    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 19:11:04.978723 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.967131    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 19:11:04.978723 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.967883    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 06 19:11:04.978723 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.976904    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 06 19:11:04.978723 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.976934    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 06 19:11:04.978723 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.976955    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 06 19:11:04.978723 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.976961    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 06 19:11:04.978723 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977014    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 19:11:04.978723 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977019    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 19:11:04.978723 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977022    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 19:11:04.978723 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977025    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 19:11:04.978723 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977028    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 19:11:04.978723 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977030    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 19:11:04.979134 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977033    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 19:11:04.979134 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977036    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 19:11:04.979134 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977038    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 19:11:04.979134 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977041    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 19:11:04.979134 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977044    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 19:11:04.979134 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977047    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 19:11:04.979134 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977050    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 19:11:04.979134 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977052    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 19:11:04.979134 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977055    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 19:11:04.979134 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977058    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 19:11:04.979134 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977062    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 19:11:04.979134 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977065    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 19:11:04.979134 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977069    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 19:11:04.979134 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977073    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 19:11:04.979134 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977076    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 19:11:04.979134 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977079    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 19:11:04.979134 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977082    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 19:11:04.979134 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977085    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 19:11:04.979134 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977088    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 19:11:04.979597 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977090    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 19:11:04.979597 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977093    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 19:11:04.979597 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977096    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 19:11:04.979597 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977098    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 19:11:04.979597 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977102    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 19:11:04.979597 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977104    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 19:11:04.979597 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977107    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 19:11:04.979597 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977110    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 19:11:04.979597 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977112    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 19:11:04.979597 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977115    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 19:11:04.979597 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977118    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 19:11:04.979597 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977120    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 19:11:04.979597 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977123    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 19:11:04.979597 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977125    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 19:11:04.979597 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977128    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 19:11:04.979597 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977130    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 19:11:04.979597 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977133    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 19:11:04.979597 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977135    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 19:11:04.979597 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977138    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 19:11:04.979597 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977140    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 19:11:04.980099 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977143    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 19:11:04.980099 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977146    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 19:11:04.980099 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977148    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 19:11:04.980099 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977151    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 19:11:04.980099 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977153    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 19:11:04.980099 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977155    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 19:11:04.980099 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977158    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 19:11:04.980099 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977160    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 19:11:04.980099 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977163    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 19:11:04.980099 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977165    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 19:11:04.980099 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977168    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 19:11:04.980099 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977170    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 19:11:04.980099 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977173    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 19:11:04.980099 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977175    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 19:11:04.980099 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977177    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 19:11:04.980099 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977181    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 19:11:04.980099 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977185    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 19:11:04.980099 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977188    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 19:11:04.980099 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977190    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 19:11:04.980099 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977193    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 19:11:04.980575 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977195    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 19:11:04.980575 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977198    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 19:11:04.980575 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977200    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 19:11:04.980575 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977202    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 19:11:04.980575 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977205    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 19:11:04.980575 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977207    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 19:11:04.980575 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977210    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 19:11:04.980575 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977212    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 19:11:04.980575 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977214    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 19:11:04.980575 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977217    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 19:11:04.980575 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977219    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 19:11:04.980575 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977222    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 19:11:04.980575 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977224    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 19:11:04.980575 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977227    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 19:11:04.980575 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977230    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 19:11:04.980575 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977233    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 19:11:04.980575 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977235    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 19:11:04.980575 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977238    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 19:11:04.980575 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977240    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 19:11:04.981060 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977243    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 19:11:04.981060 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977245    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 19:11:04.981060 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977248    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 19:11:04.981060 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977250    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 19:11:04.981060 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977253    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 19:11:04.981060 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977255    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 19:11:04.981060 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977258    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 19:11:04.981060 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977261    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 19:11:04.981060 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977263    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 19:11:04.981060 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977266    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 19:11:04.981060 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977269    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 19:11:04.981060 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977271    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 19:11:04.981060 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977274    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 19:11:04.981060 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977277    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 19:11:04.981060 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977279    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 19:11:04.981421 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.977284    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 06 19:11:04.981421 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977386    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 19:11:04.981421 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977391    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 19:11:04.981421 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977394    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 19:11:04.981421 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977397    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 19:11:04.981421 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977399    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 19:11:04.981421 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977402    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 19:11:04.981421 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977404    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 19:11:04.981421 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977407    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 19:11:04.981421 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977409    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 19:11:04.981421 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977412    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 19:11:04.981421 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977414    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 19:11:04.981421 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977417    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 19:11:04.981421 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977420    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 19:11:04.981421 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977423    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 19:11:04.981780 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977428    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 19:11:04.981780 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977431    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 19:11:04.981780 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977434    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 19:11:04.981780 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977437    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 19:11:04.981780 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977440    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 19:11:04.981780 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977442    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 19:11:04.981780 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977445    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 19:11:04.981780 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977447    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 19:11:04.981780 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977450    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 19:11:04.981780 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977453    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 19:11:04.981780 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977456    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 19:11:04.981780 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977458    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 19:11:04.981780 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977461    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 19:11:04.981780 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977464    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 19:11:04.981780 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977467    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 19:11:04.981780 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977469    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 19:11:04.981780 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977472    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 19:11:04.981780 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977475    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 19:11:04.981780 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977478    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 19:11:04.982245 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977480    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 19:11:04.982245 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977483    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 19:11:04.982245 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977486    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 19:11:04.982245 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977488    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 19:11:04.982245 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977491    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 19:11:04.982245 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977494    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 19:11:04.982245 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977497    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 19:11:04.982245 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977500    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 19:11:04.982245 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977502    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 19:11:04.982245 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977504    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 19:11:04.982245 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977507    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 19:11:04.982245 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977509    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 19:11:04.982245 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977512    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 19:11:04.982245 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977514    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 19:11:04.982245 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977517    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 19:11:04.982245 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977519    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 19:11:04.982245 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977522    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 19:11:04.982245 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977524    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 19:11:04.982245 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977526    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 19:11:04.982245 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977529    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 19:11:04.982711 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977531    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 19:11:04.982711 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977534    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 19:11:04.982711 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977536    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 19:11:04.982711 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977538    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 19:11:04.982711 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977541    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 19:11:04.982711 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977543    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 19:11:04.982711 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977546    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 19:11:04.982711 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977548    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 19:11:04.982711 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977550    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 19:11:04.982711 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977553    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 19:11:04.982711 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977555    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 19:11:04.982711 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977557    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 19:11:04.982711 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977560    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 19:11:04.982711 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977562    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 19:11:04.982711 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977564    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 19:11:04.982711 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977567    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 19:11:04.982711 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977569    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 19:11:04.982711 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977571    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 19:11:04.982711 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977574    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 19:11:04.982711 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977577    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 19:11:04.983214 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977579    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 19:11:04.983214 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977581    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 19:11:04.983214 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977584    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 19:11:04.983214 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977586    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 19:11:04.983214 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977589    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 19:11:04.983214 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977593    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 19:11:04.983214 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977596    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 19:11:04.983214 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977599    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 19:11:04.983214 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977601    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 19:11:04.983214 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977604    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 19:11:04.983214 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977606    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 19:11:04.983214 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977609    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 19:11:04.983214 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977612    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 19:11:04.983214 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977614    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 19:11:04.983214 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977616    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 19:11:04.983214 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977619    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 19:11:04.983214 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977621    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 19:11:04.983214 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977623    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 19:11:04.983214 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977626    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 19:11:04.983663 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977628    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 19:11:04.983663 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977630    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 19:11:04.983663 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977633    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 19:11:04.983663 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977636    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 19:11:04.983663 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977638    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 19:11:04.983663 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977640    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 19:11:04.983663 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:04.977643    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 19:11:04.983663 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.977648    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 06 19:11:04.983663 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.977741    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 06 19:11:04.983663 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.982151    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 06 19:11:04.983663 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.983224    2566 server.go:1019] "Starting client certificate rotation"
Sep 06 19:11:04.983663 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.983330    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 06 19:11:04.983663 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.983362    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 06 19:11:04.986736 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.986722    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 06 19:11:04.988916 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:04.988898    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 06 19:11:05.000698 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.000683    2566 log.go:25] "Validated CRI v1 runtime API"
Sep 06 19:11:05.004945 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.004910    2566 log.go:25] "Validated CRI v1 image API"
Sep 06 19:11:05.005804 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.005782    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 06 19:11:05.010276 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.010253    2566 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a0aaff79-93a7-4598-b567-9d98f956c6e1:/dev/nvme0n1p4 b217100a-c921-493f-b8a8-4d354f5e695d:/dev/nvme0n1p3]
Sep 06 19:11:05.010354 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.010275    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 06 19:11:05.014466 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.014450    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 06 19:11:05.017493 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.017278    2566 manager.go:216] Machine: {Timestamp:2026-09-06 19:11:05.015450943 +0000 UTC m=+0.381313192 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097294 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f654e5011cbfbdaece850c7cc64a9 SystemUUID:ec2f654e-5011-cbfb-daec-e850c7cc64a9 BootID:c9712bbd-1306-4fd8-b0c3-29a2c9c4fe61 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8b:91:11:7a:13 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8b:91:11:7a:13 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:6b:78:1d:d3:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 06 19:11:05.017637 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.017624    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 06 19:11:05.017764 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.017739    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 06 19:11:05.021274 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.021244    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 06 19:11:05.021404 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.021276    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-252.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 06 19:11:05.022117 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.022107    2566 topology_manager.go:138] "Creating topology manager with none policy"
Sep 06 19:11:05.022152 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.022119    2566 container_manager_linux.go:309] "Creating device plugin manager"
Sep 06 19:11:05.022152 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.022127    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 06 19:11:05.022954 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.022943    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 06 19:11:05.024234 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.024225    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 06 19:11:05.028965 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.028954    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 06 19:11:05.029074 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.029065    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 06 19:11:05.029109 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.029094    2566 kubelet.go:486] "Attempting to sync node with API server"
Sep 06 19:11:05.029109 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.029105    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 06 19:11:05.029161 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.029125    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 06 19:11:05.029161 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.029138    2566 kubelet.go:398] "Adding apiserver pod source"
Sep 06 19:11:05.029161 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.029147    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 06 19:11:05.029483 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.029472    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 06 19:11:05.029545 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.029489    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 06 19:11:05.031828 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.031806    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 06 19:11:05.034795 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.034782    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 06 19:11:05.034837 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.034812    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 06 19:11:05.036175 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.036163    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 06 19:11:05.036241 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.036179    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 06 19:11:05.036241 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.036185    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 06 19:11:05.036241 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.036190    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 06 19:11:05.036241 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.036200    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 06 19:11:05.036241 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.036206    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 06 19:11:05.036241 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.036203    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lxckj"
Sep 06 19:11:05.036241 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.036218    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 06 19:11:05.036241 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.036224    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 06 19:11:05.036241 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.036231    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 06 19:11:05.036241 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.036236    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 06 19:11:05.036241 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.036243    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 06 19:11:05.036527 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.036249    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 06 19:11:05.037205 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.037193    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 06 19:11:05.037245 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.037208    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 06 19:11:05.041502 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.041454    2566 server.go:1268] "Started kubelet"
Sep 06 19:11:05.041610 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.041566    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 06 19:11:05.041650 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.041615    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 06 19:11:05.041650 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.041635    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 06 19:11:05.041793 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:05.041771    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 06 19:11:05.042213 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:05.041908    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-252.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 06 19:11:05.042213 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.041594    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-252.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 06 19:11:05.042456 ip-10-0-129-252 systemd[1]: Started Kubernetes Kubelet.
Sep 06 19:11:05.042978 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.042963    2566 server.go:310] "Adding debug handlers to kubelet server"
Sep 06 19:11:05.043091 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.042972    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lxckj"
Sep 06 19:11:05.043281 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.043261    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 06 19:11:05.045859 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:05.045841    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 06 19:11:05.047106 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:05.046001    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-252.ec2.internal.18d2d10f27c4c34c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-252.ec2.internal,UID:ip-10-0-129-252.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-252.ec2.internal,},FirstTimestamp:2026-09-06 19:11:05.041429324 +0000 UTC m=+0.407291572,LastTimestamp:2026-09-06 19:11:05.041429324 +0000 UTC m=+0.407291572,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-252.ec2.internal,}"
Sep 06 19:11:05.049652 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.049628    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 06 19:11:05.050353 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.050340    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 06 19:11:05.051152 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:05.051001    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-252.ec2.internal\" not found"
Sep 06 19:11:05.051152 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.051099    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 06 19:11:05.051271 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.051102    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 06 19:11:05.051271 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.051170    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 06 19:11:05.051271 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.051220    2566 factory.go:55] Registering systemd factory
Sep 06 19:11:05.051271 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.051232    2566 factory.go:223] Registration of the systemd container factory successfully
Sep 06 19:11:05.051271 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.051248    2566 reconstruct.go:97] "Volume reconstruction finished"
Sep 06 19:11:05.051271 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.051259    2566 reconciler.go:29] "Reconciler: start to sync state"
Sep 06 19:11:05.051454 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.051428    2566 factory.go:153] Registering CRI-O factory
Sep 06 19:11:05.051454 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.051441    2566 factory.go:223] Registration of the crio container factory successfully
Sep 06 19:11:05.051512 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.051488    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 06 19:11:05.051541 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.051505    2566 factory.go:103] Registering Raw factory
Sep 06 19:11:05.051541 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.051538    2566 manager.go:1196] Started watching for new ooms in manager
Sep 06 19:11:05.051885 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.051872    2566 manager.go:349] Starting recovery of all containers
Sep 06 19:11:05.059165 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.059142    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 19:11:05.061387 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:05.061365    2566 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-252.ec2.internal\" not found" node="ip-10-0-129-252.ec2.internal"
Sep 06 19:11:05.063561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.063543    2566 manager.go:354] Recovery completed
Sep 06 19:11:05.067951 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.067938    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 19:11:05.069614 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.069601    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 19:11:05.069671 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.069627    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 19:11:05.069671 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.069637    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-252.ec2.internal" event="NodeHasSufficientPID"
Sep 06 19:11:05.070150 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.070136    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 06 19:11:05.070150 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.070149    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 19:11:05.070266 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.070152    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 06 19:11:05.070266 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.070194    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 06 19:11:05.070795 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.070781    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 19:11:05.070838 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.070810    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 19:11:05.070838 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.070822    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-252.ec2.internal" event="NodeHasSufficientPID"
Sep 06 19:11:05.072346 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.072332    2566 policy_none.go:49] "None policy: Start"
Sep 06 19:11:05.072418 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.072352    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 06 19:11:05.072418 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.072366    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 06 19:11:05.074435 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.074420    2566 policy_none.go:47] "Start"
Sep 06 19:11:05.101578 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.101560    2566 manager.go:337] "Starting Device Plugin manager"
Sep 06 19:11:05.128143 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:05.101613    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 06 19:11:05.128143 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.101627    2566 server.go:87] "Starting device plugin registration server"
Sep 06 19:11:05.128143 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.101985    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 06 19:11:05.128143 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.101998    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 06 19:11:05.128143 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.102089    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 06 19:11:05.128143 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.102170    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 06 19:11:05.128143 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.102180    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 06 19:11:05.128143 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:05.102637    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 06 19:11:05.128143 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:05.102686    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-252.ec2.internal\" not found"
Sep 06 19:11:05.128143 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.117642    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 06 19:11:05.128143 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.118915    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 06 19:11:05.128143 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.118971    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 06 19:11:05.128143 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.118991    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 06 19:11:05.128143 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:05.119028    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 06 19:11:05.128143 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.120656    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 19:11:05.202219 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.202189    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 19:11:05.203534 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.203511    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 19:11:05.203612 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.203546    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 19:11:05.203612 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.203558    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-252.ec2.internal" event="NodeHasSufficientPID"
Sep 06 19:11:05.203612 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.203580    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-252.ec2.internal"
Sep 06 19:11:05.212999 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.212940    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-252.ec2.internal"
Sep 06 19:11:05.212999 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:05.212962    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-252.ec2.internal\": node \"ip-10-0-129-252.ec2.internal\" not found"
Sep 06 19:11:05.219938 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.219906    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-252.ec2.internal"]
Sep 06 19:11:05.219991 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.219979    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 19:11:05.223743 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.223726    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 19:11:05.223838 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.223752    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 19:11:05.223838 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.223764    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-252.ec2.internal" event="NodeHasSufficientPID"
Sep 06 19:11:05.226082 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.226070    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 19:11:05.226221 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.226205    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.ec2.internal"
Sep 06 19:11:05.226273 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.226238    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 19:11:05.226307 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:05.226296    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-252.ec2.internal\" not found"
Sep 06 19:11:05.226783 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.226759    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 19:11:05.226783 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.226783    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 19:11:05.226905 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.226799    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 19:11:05.226905 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.226808    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-252.ec2.internal" event="NodeHasSufficientPID"
Sep 06 19:11:05.226905 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.226784    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 19:11:05.226905 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.226834    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-252.ec2.internal" event="NodeHasSufficientPID"
Sep 06 19:11:05.229142 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.229126    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.ec2.internal"
Sep 06 19:11:05.229183 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.229159    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 19:11:05.229868 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.229852    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 19:11:05.229957 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.229876    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 19:11:05.229957 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.229885    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-252.ec2.internal" event="NodeHasSufficientPID"
Sep 06 19:11:05.256457 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:05.256437    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-252.ec2.internal\" not found" node="ip-10-0-129-252.ec2.internal"
Sep 06 19:11:05.259816 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:05.259802    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-252.ec2.internal\" not found" node="ip-10-0-129-252.ec2.internal"
Sep 06 19:11:05.326687 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:05.326667    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-252.ec2.internal\" not found"
Sep 06 19:11:05.352933 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.352899    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a6b312538c38d2cf66321b95dbc03a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-252.ec2.internal\" (UID: \"1a6b312538c38d2cf66321b95dbc03a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.ec2.internal"
Sep 06 19:11:05.353011 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.352935    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a6b312538c38d2cf66321b95dbc03a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-252.ec2.internal\" (UID: \"1a6b312538c38d2cf66321b95dbc03a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.ec2.internal"
Sep 06 19:11:05.353011 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.352951    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/faae1a37c3a006a83d2995caac1d3848-config\") pod \"kube-apiserver-proxy-ip-10-0-129-252.ec2.internal\" (UID: \"faae1a37c3a006a83d2995caac1d3848\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.ec2.internal"
Sep 06 19:11:05.427043 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:05.427016    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-252.ec2.internal\" not found"
Sep 06 19:11:05.453430 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.453412    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a6b312538c38d2cf66321b95dbc03a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-252.ec2.internal\" (UID: \"1a6b312538c38d2cf66321b95dbc03a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.ec2.internal"
Sep 06 19:11:05.453726 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.453328    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a6b312538c38d2cf66321b95dbc03a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-252.ec2.internal\" (UID: \"1a6b312538c38d2cf66321b95dbc03a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.ec2.internal"
Sep 06 19:11:05.453765 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.453736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a6b312538c38d2cf66321b95dbc03a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-252.ec2.internal\" (UID: \"1a6b312538c38d2cf66321b95dbc03a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.ec2.internal"
Sep 06 19:11:05.453765 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.453753    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/faae1a37c3a006a83d2995caac1d3848-config\") pod \"kube-apiserver-proxy-ip-10-0-129-252.ec2.internal\" (UID: \"faae1a37c3a006a83d2995caac1d3848\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.ec2.internal"
Sep 06 19:11:05.453835 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.453774    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/faae1a37c3a006a83d2995caac1d3848-config\") pod \"kube-apiserver-proxy-ip-10-0-129-252.ec2.internal\" (UID: \"faae1a37c3a006a83d2995caac1d3848\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.ec2.internal"
Sep 06 19:11:05.453835 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.453796    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a6b312538c38d2cf66321b95dbc03a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-252.ec2.internal\" (UID: \"1a6b312538c38d2cf66321b95dbc03a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.ec2.internal"
Sep 06 19:11:05.527652 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:05.527598    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-252.ec2.internal\" not found"
Sep 06 19:11:05.559243 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.559228    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.ec2.internal"
Sep 06 19:11:05.564618 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.564511    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.ec2.internal"
Sep 06 19:11:05.628688 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:05.628666    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-252.ec2.internal\" not found"
Sep 06 19:11:05.729217 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:05.729183    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-252.ec2.internal\" not found"
Sep 06 19:11:05.829708 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:05.829638    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-252.ec2.internal\" not found"
Sep 06 19:11:05.930158 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:05.930133    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-252.ec2.internal\" not found"
Sep 06 19:11:05.982610 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.982588    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 06 19:11:05.983013 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.982716    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 06 19:11:05.983013 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:05.982742    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 06 19:11:06.031014 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:06.030988    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-252.ec2.internal\" not found"
Sep 06 19:11:06.049293 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:06.049255    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-05 19:06:06 +0000 UTC" deadline="2028-04-05 03:37:54.662782575 +0000 UTC"
Sep 06 19:11:06.049293 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:06.049285    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13832h26m48.613501403s"
Sep 06 19:11:06.131507 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:06.131447    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-252.ec2.internal\" not found"
Sep 06 19:11:06.169359 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:06.169326    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaae1a37c3a006a83d2995caac1d3848.slice/crio-cbfdcc6c880713affeebca35223b442d45a541da2b68b00a919fef363823f66f WatchSource:0}: Error finding container cbfdcc6c880713affeebca35223b442d45a541da2b68b00a919fef363823f66f: Status 404 returned error can't find the container with id cbfdcc6c880713affeebca35223b442d45a541da2b68b00a919fef363823f66f
Sep 06 19:11:06.169556 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:06.169541    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6b312538c38d2cf66321b95dbc03a9.slice/crio-668daa2dd992c7548722357006ca1eae6f455dfc78e132698c73956df6ddb687 WatchSource:0}: Error finding container 668daa2dd992c7548722357006ca1eae6f455dfc78e132698c73956df6ddb687: Status 404 returned error can't find the container with id 668daa2dd992c7548722357006ca1eae6f455dfc78e132698c73956df6ddb687
Sep 06 19:11:06.174476 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:06.174457    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 19:11:06.185573 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:06.185553    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 19:11:06.251552 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:06.251528    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.ec2.internal"
Sep 06 19:11:06.261308 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:06.261291    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 06 19:11:06.263342 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:06.263329    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.ec2.internal"
Sep 06 19:11:06.270897 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:06.270884    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 06 19:11:06.373973 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:06.373942    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 19:11:08.178325 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.178243    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 19:11:08.215166 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.215141    2566 apiserver.go:52] "Watching apiserver"
Sep 06 19:11:08.225147 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.225116    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 06 19:11:08.225547 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.225518    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-s9h5h","openshift-ovn-kubernetes/ovnkube-node-gnssq","kube-system/global-pull-secret-syncer-7crp9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs","openshift-dns/node-resolver-jm5gt","openshift-image-registry/node-ca-x75tt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.ec2.internal","openshift-multus/network-metrics-daemon-qlvsw","openshift-network-operator/iptables-alerter-jdfqm","kube-system/konnectivity-agent-b5vzh","kube-system/kube-apiserver-proxy-ip-10-0-129-252.ec2.internal","openshift-cluster-node-tuning-operator/tuned-8hj5v","openshift-multus/multus-additional-cni-plugins-q4ncl","openshift-multus/multus-tb692"]
Sep 06 19:11:08.228140 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.228117    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b5vzh"
Sep 06 19:11:08.230791 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.230766    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-57bcl\""
Sep 06 19:11:08.230898 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.230823    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 06 19:11:08.230956 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.230767    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 06 19:11:08.231126 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.231106    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q4ncl"
Sep 06 19:11:08.233391 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.233375    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:08.233480 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:08.233436    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7crp9" podUID="dcd6e3bb-3d92-4fe9-8432-62ef628b2472"
Sep 06 19:11:08.234067 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.234045    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 06 19:11:08.234184 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.234070    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 06 19:11:08.234184 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.234082    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-crhzb\""
Sep 06 19:11:08.234300 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.234242    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 06 19:11:08.234300 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.234256    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 06 19:11:08.234391 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.234342    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 06 19:11:08.235463 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.235443    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 06 19:11:08.236508 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.236491    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs"
Sep 06 19:11:08.239357 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.238523    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jm5gt"
Sep 06 19:11:08.240879 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.240770    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 06 19:11:08.241000 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.240982    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 06 19:11:08.241251 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.241227    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 06 19:11:08.241415 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.241402    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wpgkz\""
Sep 06 19:11:08.241754 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.241648    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 06 19:11:08.241754 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.241741    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x75tt"
Sep 06 19:11:08.241909 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.241857    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 06 19:11:08.241956 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.241744    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4ztjq\""
Sep 06 19:11:08.243812 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.243763    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jdfqm"
Sep 06 19:11:08.245961 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.244763    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 06 19:11:08.245961 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.244906    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 06 19:11:08.245961 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.245064    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-x5qlk\""
Sep 06 19:11:08.246156 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.246025    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 06 19:11:08.249199 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.246477    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 06 19:11:08.249199 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.246488    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b554b5b-8ce8-4674-b7d3-235a7c0b777b-cnibin\") pod \"multus-additional-cni-plugins-q4ncl\" (UID: \"0b554b5b-8ce8-4674-b7d3-235a7c0b777b\") " pod="openshift-multus/multus-additional-cni-plugins-q4ncl"
Sep 06 19:11:08.249199 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.246523    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-dbus\") pod \"global-pull-secret-syncer-7crp9\" (UID: \"dcd6e3bb-3d92-4fe9-8432-62ef628b2472\") " pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:08.249199 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.246551    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-original-pull-secret\") pod \"global-pull-secret-syncer-7crp9\" (UID: \"dcd6e3bb-3d92-4fe9-8432-62ef628b2472\") " pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:08.249199 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.246565    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 06 19:11:08.249199 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.246584    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4557de3f-1c4f-448f-aec4-4552db157e3e-socket-dir\") pod \"aws-ebs-csi-driver-node-r4ggs\" (UID: \"4557de3f-1c4f-448f-aec4-4552db157e3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs"
Sep 06 19:11:08.249199 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.246698    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/176abf0b-9b68-46d3-a657-ab219939488a-hosts-file\") pod \"node-resolver-jm5gt\" (UID: \"176abf0b-9b68-46d3-a657-ab219939488a\") " pod="openshift-dns/node-resolver-jm5gt"
Sep 06 19:11:08.249199 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.246731    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df58r\" (UniqueName: \"kubernetes.io/projected/176abf0b-9b68-46d3-a657-ab219939488a-kube-api-access-df58r\") pod \"node-resolver-jm5gt\" (UID: \"176abf0b-9b68-46d3-a657-ab219939488a\") " pod="openshift-dns/node-resolver-jm5gt"
Sep 06 19:11:08.249199 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.246760    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/30fe2339-bf82-4be5-8cf4-f68f98014b4f-konnectivity-ca\") pod \"konnectivity-agent-b5vzh\" (UID: \"30fe2339-bf82-4be5-8cf4-f68f98014b4f\") " pod="kube-system/konnectivity-agent-b5vzh"
Sep 06 19:11:08.249199 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.246818    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b554b5b-8ce8-4674-b7d3-235a7c0b777b-system-cni-dir\") pod \"multus-additional-cni-plugins-q4ncl\" (UID: \"0b554b5b-8ce8-4674-b7d3-235a7c0b777b\") " pod="openshift-multus/multus-additional-cni-plugins-q4ncl"
Sep 06 19:11:08.249199 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.246851    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0b554b5b-8ce8-4674-b7d3-235a7c0b777b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q4ncl\" (UID: \"0b554b5b-8ce8-4674-b7d3-235a7c0b777b\") " pod="openshift-multus/multus-additional-cni-plugins-q4ncl"
Sep 06 19:11:08.249199 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.246879    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4557de3f-1c4f-448f-aec4-4552db157e3e-etc-selinux\") pod \"aws-ebs-csi-driver-node-r4ggs\" (UID: \"4557de3f-1c4f-448f-aec4-4552db157e3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs"
Sep 06 19:11:08.249199 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.247025    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pgb29\""
Sep 06 19:11:08.249199 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.246907    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4557de3f-1c4f-448f-aec4-4552db157e3e-sys-fs\") pod \"aws-ebs-csi-driver-node-r4ggs\" (UID: \"4557de3f-1c4f-448f-aec4-4552db157e3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs"
Sep 06 19:11:08.249199 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.247071    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176abf0b-9b68-46d3-a657-ab219939488a-tmp-dir\") pod \"node-resolver-jm5gt\" (UID: \"176abf0b-9b68-46d3-a657-ab219939488a\") " pod="openshift-dns/node-resolver-jm5gt"
Sep 06 19:11:08.249199 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.247100    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0b554b5b-8ce8-4674-b7d3-235a7c0b777b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q4ncl\" (UID: \"0b554b5b-8ce8-4674-b7d3-235a7c0b777b\") " pod="openshift-multus/multus-additional-cni-plugins-q4ncl"
Sep 06 19:11:08.249199 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.247134    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24wl8\" (UniqueName: \"kubernetes.io/projected/0b554b5b-8ce8-4674-b7d3-235a7c0b777b-kube-api-access-24wl8\") pod \"multus-additional-cni-plugins-q4ncl\" (UID: \"0b554b5b-8ce8-4674-b7d3-235a7c0b777b\") " pod="openshift-multus/multus-additional-cni-plugins-q4ncl"
Sep 06 19:11:08.249199 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.247270    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 06 19:11:08.250050 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.247386    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 06 19:11:08.250050 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.247166    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4557de3f-1c4f-448f-aec4-4552db157e3e-device-dir\") pod \"aws-ebs-csi-driver-node-r4ggs\" (UID: \"4557de3f-1c4f-448f-aec4-4552db157e3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs"
Sep 06 19:11:08.250050 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.247480    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/30fe2339-bf82-4be5-8cf4-f68f98014b4f-agent-certs\") pod \"konnectivity-agent-b5vzh\" (UID: \"30fe2339-bf82-4be5-8cf4-f68f98014b4f\") " pod="kube-system/konnectivity-agent-b5vzh"
Sep 06 19:11:08.250050 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.247511    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b554b5b-8ce8-4674-b7d3-235a7c0b777b-cni-binary-copy\") pod \"multus-additional-cni-plugins-q4ncl\" (UID: \"0b554b5b-8ce8-4674-b7d3-235a7c0b777b\") " pod="openshift-multus/multus-additional-cni-plugins-q4ncl"
Sep 06 19:11:08.250050 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.247542    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b554b5b-8ce8-4674-b7d3-235a7c0b777b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q4ncl\" (UID: \"0b554b5b-8ce8-4674-b7d3-235a7c0b777b\") " pod="openshift-multus/multus-additional-cni-plugins-q4ncl"
Sep 06 19:11:08.250050 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.247597    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-kubelet-config\") pod \"global-pull-secret-syncer-7crp9\" (UID: \"dcd6e3bb-3d92-4fe9-8432-62ef628b2472\") " pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:08.250050 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.247650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4557de3f-1c4f-448f-aec4-4552db157e3e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r4ggs\" (UID: \"4557de3f-1c4f-448f-aec4-4552db157e3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs"
Sep 06 19:11:08.250050 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.247710    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4557de3f-1c4f-448f-aec4-4552db157e3e-registration-dir\") pod \"aws-ebs-csi-driver-node-r4ggs\" (UID: \"4557de3f-1c4f-448f-aec4-4552db157e3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs"
Sep 06 19:11:08.250050 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.247740    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.250050 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.247749    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhgf7\" (UniqueName: \"kubernetes.io/projected/4557de3f-1c4f-448f-aec4-4552db157e3e-kube-api-access-mhgf7\") pod \"aws-ebs-csi-driver-node-r4ggs\" (UID: \"4557de3f-1c4f-448f-aec4-4552db157e3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs"
Sep 06 19:11:08.250730 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.250464    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 06 19:11:08.250730 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.250551    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6dkxz\""
Sep 06 19:11:08.250902 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.250868    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.253350 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.253313    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 06 19:11:08.254343 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.254325    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 06 19:11:08.254462 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.254443    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nkn8c\""
Sep 06 19:11:08.254553 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.254395    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 06 19:11:08.254635 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.254452    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 06 19:11:08.254696 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.254390    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 06 19:11:08.254958 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.254922    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 06 19:11:08.255085 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.254837    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:08.255711 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:08.255688    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9h5h" podUID="52bd7358-7476-41f4-95da-5bdfefff01db"
Sep 06 19:11:08.258145 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.258129    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:08.258233 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:08.258184    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlvsw" podUID="83a7379b-ab58-4439-998e-448e48150f3b"
Sep 06 19:11:08.260460 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.259956    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.262536 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.262514    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 06 19:11:08.262989 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.262973    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 06 19:11:08.263147 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.262975    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8wxbg\""
Sep 06 19:11:08.267709 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.267689    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vh8kv"
Sep 06 19:11:08.276486 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.276465    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vh8kv"
Sep 06 19:11:08.309828 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.309786    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.ec2.internal" event={"ID":"faae1a37c3a006a83d2995caac1d3848","Type":"ContainerStarted","Data":"cbfdcc6c880713affeebca35223b442d45a541da2b68b00a919fef363823f66f"}
Sep 06 19:11:08.313055 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.313032    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.ec2.internal" event={"ID":"1a6b312538c38d2cf66321b95dbc03a9","Type":"ContainerStarted","Data":"668daa2dd992c7548722357006ca1eae6f455dfc78e132698c73956df6ddb687"}
Sep 06 19:11:08.336866 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.336848    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 06 19:11:08.348463 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348439    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a68be042-a345-44a5-9564-1d6d16d11b8a-serviceca\") pod \"node-ca-x75tt\" (UID: \"a68be042-a345-44a5-9564-1d6d16d11b8a\") " pod="openshift-image-registry/node-ca-x75tt"
Sep 06 19:11:08.348575 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348477    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.348575 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348509    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24wl8\" (UniqueName: \"kubernetes.io/projected/0b554b5b-8ce8-4674-b7d3-235a7c0b777b-kube-api-access-24wl8\") pod \"multus-additional-cni-plugins-q4ncl\" (UID: \"0b554b5b-8ce8-4674-b7d3-235a7c0b777b\") " pod="openshift-multus/multus-additional-cni-plugins-q4ncl"
Sep 06 19:11:08.348575 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-multus-cni-dir\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.348575 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348555    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-host-run-netns\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.348575 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348574    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-run-openvswitch\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.348809 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348589    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-kubelet-config\") pod \"global-pull-secret-syncer-7crp9\" (UID: \"dcd6e3bb-3d92-4fe9-8432-62ef628b2472\") " pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:08.348809 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348621    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-run-systemd\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.348809 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348642    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9254e6f1-b73d-49cc-8bc1-513925ccde4f-ovnkube-config\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.348809 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348656    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99gc7\" (UniqueName: \"kubernetes.io/projected/9254e6f1-b73d-49cc-8bc1-513925ccde4f-kube-api-access-99gc7\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.348809 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348671    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-dbus\") pod \"global-pull-secret-syncer-7crp9\" (UID: \"dcd6e3bb-3d92-4fe9-8432-62ef628b2472\") " pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:08.348809 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348686    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-original-pull-secret\") pod \"global-pull-secret-syncer-7crp9\" (UID: \"dcd6e3bb-3d92-4fe9-8432-62ef628b2472\") " pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:08.348809 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348699    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-host-var-lib-kubelet\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.348809 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348714    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a68be042-a345-44a5-9564-1d6d16d11b8a-host\") pod \"node-ca-x75tt\" (UID: \"a68be042-a345-44a5-9564-1d6d16d11b8a\") " pod="openshift-image-registry/node-ca-x75tt"
Sep 06 19:11:08.348809 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348727    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-node-log\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.348809 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348740    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-log-socket\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.348809 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348761    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9254e6f1-b73d-49cc-8bc1-513925ccde4f-ovn-node-metrics-cert\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.348809 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348794    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0b554b5b-8ce8-4674-b7d3-235a7c0b777b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q4ncl\" (UID: \"0b554b5b-8ce8-4674-b7d3-235a7c0b777b\") " pod="openshift-multus/multus-additional-cni-plugins-q4ncl"
Sep 06 19:11:08.348809 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348811    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176abf0b-9b68-46d3-a657-ab219939488a-tmp-dir\") pod \"node-resolver-jm5gt\" (UID: \"176abf0b-9b68-46d3-a657-ab219939488a\") " pod="openshift-dns/node-resolver-jm5gt"
Sep 06 19:11:08.349254 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348826    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-etc-sysctl-d\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.349254 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348840    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-lib-modules\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.349254 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348854    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-host-var-lib-cni-multus\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.349254 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348869    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65fng\" (UniqueName: \"kubernetes.io/projected/83a7379b-ab58-4439-998e-448e48150f3b-kube-api-access-65fng\") pod \"network-metrics-daemon-qlvsw\" (UID: \"83a7379b-ab58-4439-998e-448e48150f3b\") " pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:08.349254 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348894    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0b554b5b-8ce8-4674-b7d3-235a7c0b777b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q4ncl\" (UID: \"0b554b5b-8ce8-4674-b7d3-235a7c0b777b\") " pod="openshift-multus/multus-additional-cni-plugins-q4ncl"
Sep 06 19:11:08.349254 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348919    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-etc-tuned\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.349254 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348941    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9254e6f1-b73d-49cc-8bc1-513925ccde4f-ovnkube-script-lib\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.349254 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348963    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs\") pod \"network-metrics-daemon-qlvsw\" (UID: \"83a7379b-ab58-4439-998e-448e48150f3b\") " pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:08.349254 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348981    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b554b5b-8ce8-4674-b7d3-235a7c0b777b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q4ncl\" (UID: \"0b554b5b-8ce8-4674-b7d3-235a7c0b777b\") " pod="openshift-multus/multus-additional-cni-plugins-q4ncl"
Sep 06 19:11:08.349254 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.348994    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-sys\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.349254 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.349025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/317e1dba-5d84-4cfa-84aa-7d1a3dc399d5-iptables-alerter-script\") pod \"iptables-alerter-jdfqm\" (UID: \"317e1dba-5d84-4cfa-84aa-7d1a3dc399d5\") " pod="openshift-network-operator/iptables-alerter-jdfqm"
Sep 06 19:11:08.349254 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.349048    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-host-cni-netd\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.349254 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.349072    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/176abf0b-9b68-46d3-a657-ab219939488a-hosts-file\") pod \"node-resolver-jm5gt\" (UID: \"176abf0b-9b68-46d3-a657-ab219939488a\") " pod="openshift-dns/node-resolver-jm5gt"
Sep 06 19:11:08.349254 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.349112    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-system-cni-dir\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.349254 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.349126    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/176abf0b-9b68-46d3-a657-ab219939488a-hosts-file\") pod \"node-resolver-jm5gt\" (UID: \"176abf0b-9b68-46d3-a657-ab219939488a\") " pod="openshift-dns/node-resolver-jm5gt"
Sep 06 19:11:08.349254 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.349148    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-multus-socket-dir-parent\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.349254 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.349236    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-kubelet-config\") pod \"global-pull-secret-syncer-7crp9\" (UID: \"dcd6e3bb-3d92-4fe9-8432-62ef628b2472\") " pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:08.350025 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.349508    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b554b5b-8ce8-4674-b7d3-235a7c0b777b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q4ncl\" (UID: \"0b554b5b-8ce8-4674-b7d3-235a7c0b777b\") " pod="openshift-multus/multus-additional-cni-plugins-q4ncl"
Sep 06 19:11:08.350025 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.349562    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-dbus\") pod \"global-pull-secret-syncer-7crp9\" (UID: \"dcd6e3bb-3d92-4fe9-8432-62ef628b2472\") " pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:08.350025 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.349594    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-host-var-lib-cni-bin\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.350025 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.349630    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jllc8\" (UniqueName: \"kubernetes.io/projected/317e1dba-5d84-4cfa-84aa-7d1a3dc399d5-kube-api-access-jllc8\") pod \"iptables-alerter-jdfqm\" (UID: \"317e1dba-5d84-4cfa-84aa-7d1a3dc399d5\") " pod="openshift-network-operator/iptables-alerter-jdfqm"
Sep 06 19:11:08.350025 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.349661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4557de3f-1c4f-448f-aec4-4552db157e3e-etc-selinux\") pod \"aws-ebs-csi-driver-node-r4ggs\" (UID: \"4557de3f-1c4f-448f-aec4-4552db157e3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs"
Sep 06 19:11:08.350025 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.349682    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-var-lib-kubelet\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.350025 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.349706    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-multus-conf-dir\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.350025 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.349730    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-host-run-multus-certs\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.350025 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.349726    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0b554b5b-8ce8-4674-b7d3-235a7c0b777b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q4ncl\" (UID: \"0b554b5b-8ce8-4674-b7d3-235a7c0b777b\") " pod="openshift-multus/multus-additional-cni-plugins-q4ncl"
Sep 06 19:11:08.350025 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.349789    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176abf0b-9b68-46d3-a657-ab219939488a-tmp-dir\") pod \"node-resolver-jm5gt\" (UID: \"176abf0b-9b68-46d3-a657-ab219939488a\") " pod="openshift-dns/node-resolver-jm5gt"
Sep 06 19:11:08.350025 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:08.349813    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 19:11:08.350025 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.349813    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-run\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.350025 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.349813    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4557de3f-1c4f-448f-aec4-4552db157e3e-etc-selinux\") pod \"aws-ebs-csi-driver-node-r4ggs\" (UID: \"4557de3f-1c4f-448f-aec4-4552db157e3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs"
Sep 06 19:11:08.350025 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:08.349888    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-original-pull-secret podName:dcd6e3bb-3d92-4fe9-8432-62ef628b2472 nodeName:}" failed. No retries permitted until 2026-09-06 19:11:08.849854925 +0000 UTC m=+3.031036723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-original-pull-secret") pod "global-pull-secret-syncer-7crp9" (UID: "dcd6e3bb-3d92-4fe9-8432-62ef628b2472") : object "kube-system"/"original-pull-secret" not registered
Sep 06 19:11:08.350025 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.349936    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-tmp\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.350025 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.349973    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-var-lib-openvswitch\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.350832 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.349998    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-etc-openvswitch\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.350832 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.350050    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b554b5b-8ce8-4674-b7d3-235a7c0b777b-cni-binary-copy\") pod \"multus-additional-cni-plugins-q4ncl\" (UID: \"0b554b5b-8ce8-4674-b7d3-235a7c0b777b\") " pod="openshift-multus/multus-additional-cni-plugins-q4ncl"
Sep 06 19:11:08.350832 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.350089    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4557de3f-1c4f-448f-aec4-4552db157e3e-registration-dir\") pod \"aws-ebs-csi-driver-node-r4ggs\" (UID: \"4557de3f-1c4f-448f-aec4-4552db157e3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs"
Sep 06 19:11:08.350832 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.350146    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhgf7\" (UniqueName: \"kubernetes.io/projected/4557de3f-1c4f-448f-aec4-4552db157e3e-kube-api-access-mhgf7\") pod \"aws-ebs-csi-driver-node-r4ggs\" (UID: \"4557de3f-1c4f-448f-aec4-4552db157e3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs"
Sep 06 19:11:08.350832 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.350188    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-etc-systemd\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.350832 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.350213    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a21c0317-3ef0-4b2d-ae52-809097383928-cni-binary-copy\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.350832 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.350241    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-host-cni-bin\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.350832 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.350266    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b554b5b-8ce8-4674-b7d3-235a7c0b777b-cnibin\") pod \"multus-additional-cni-plugins-q4ncl\" (UID: \"0b554b5b-8ce8-4674-b7d3-235a7c0b777b\") " pod="openshift-multus/multus-additional-cni-plugins-q4ncl"
Sep 06 19:11:08.350832 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.350288    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4557de3f-1c4f-448f-aec4-4552db157e3e-socket-dir\") pod \"aws-ebs-csi-driver-node-r4ggs\" (UID: \"4557de3f-1c4f-448f-aec4-4552db157e3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs"
Sep 06 19:11:08.350832 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.350329    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0b554b5b-8ce8-4674-b7d3-235a7c0b777b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q4ncl\" (UID: \"0b554b5b-8ce8-4674-b7d3-235a7c0b777b\") " pod="openshift-multus/multus-additional-cni-plugins-q4ncl"
Sep 06 19:11:08.350832 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.350332    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df58r\" (UniqueName: \"kubernetes.io/projected/176abf0b-9b68-46d3-a657-ab219939488a-kube-api-access-df58r\") pod \"node-resolver-jm5gt\" (UID: \"176abf0b-9b68-46d3-a657-ab219939488a\") " pod="openshift-dns/node-resolver-jm5gt"
Sep 06 19:11:08.350832 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.350379    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-host\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.350832 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.350405    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-cnibin\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.350832 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.350405    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4557de3f-1c4f-448f-aec4-4552db157e3e-registration-dir\") pod \"aws-ebs-csi-driver-node-r4ggs\" (UID: \"4557de3f-1c4f-448f-aec4-4552db157e3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs"
Sep 06 19:11:08.350832 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.350436    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-hostroot\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.350832 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.350464    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kvr2\" (UniqueName: \"kubernetes.io/projected/52bd7358-7476-41f4-95da-5bdfefff01db-kube-api-access-5kvr2\") pod \"network-check-target-s9h5h\" (UID: \"52bd7358-7476-41f4-95da-5bdfefff01db\") " pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:08.350832 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.350540    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b554b5b-8ce8-4674-b7d3-235a7c0b777b-cnibin\") pod \"multus-additional-cni-plugins-q4ncl\" (UID: \"0b554b5b-8ce8-4674-b7d3-235a7c0b777b\") " pod="openshift-multus/multus-additional-cni-plugins-q4ncl"
Sep 06 19:11:08.351492 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.350548    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b554b5b-8ce8-4674-b7d3-235a7c0b777b-cni-binary-copy\") pod \"multus-additional-cni-plugins-q4ncl\" (UID: \"0b554b5b-8ce8-4674-b7d3-235a7c0b777b\") " pod="openshift-multus/multus-additional-cni-plugins-q4ncl"
Sep 06 19:11:08.351492 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.350542    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/317e1dba-5d84-4cfa-84aa-7d1a3dc399d5-host-slash\") pod \"iptables-alerter-jdfqm\" (UID: \"317e1dba-5d84-4cfa-84aa-7d1a3dc399d5\") " pod="openshift-network-operator/iptables-alerter-jdfqm"
Sep 06 19:11:08.351492 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.350852    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4557de3f-1c4f-448f-aec4-4552db157e3e-socket-dir\") pod \"aws-ebs-csi-driver-node-r4ggs\" (UID: \"4557de3f-1c4f-448f-aec4-4552db157e3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs"
Sep 06 19:11:08.351492 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.351025    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b554b5b-8ce8-4674-b7d3-235a7c0b777b-system-cni-dir\") pod \"multus-additional-cni-plugins-q4ncl\" (UID: \"0b554b5b-8ce8-4674-b7d3-235a7c0b777b\") " pod="openshift-multus/multus-additional-cni-plugins-q4ncl"
Sep 06 19:11:08.351492 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.351071    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4557de3f-1c4f-448f-aec4-4552db157e3e-sys-fs\") pod \"aws-ebs-csi-driver-node-r4ggs\" (UID: \"4557de3f-1c4f-448f-aec4-4552db157e3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs"
Sep 06 19:11:08.351492 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.351101    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-etc-sysconfig\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.351492 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.351127    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-host-kubelet\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.351492 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.351164    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-systemd-units\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.351492 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.351191    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-host-slash\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.351492 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.351216    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-run-ovn\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.351492 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.351244    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4557de3f-1c4f-448f-aec4-4552db157e3e-device-dir\") pod \"aws-ebs-csi-driver-node-r4ggs\" (UID: \"4557de3f-1c4f-448f-aec4-4552db157e3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs"
Sep 06 19:11:08.351492 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.351265    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-etc-kubernetes\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.351492 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.351291    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-etc-kubernetes\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.351492 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.351322    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-host-run-netns\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.351492 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.351350    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/30fe2339-bf82-4be5-8cf4-f68f98014b4f-agent-certs\") pod \"konnectivity-agent-b5vzh\" (UID: \"30fe2339-bf82-4be5-8cf4-f68f98014b4f\") " pod="kube-system/konnectivity-agent-b5vzh"
Sep 06 19:11:08.351492 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.351378    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4557de3f-1c4f-448f-aec4-4552db157e3e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r4ggs\" (UID: \"4557de3f-1c4f-448f-aec4-4552db157e3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs"
Sep 06 19:11:08.351492 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.351401    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj8q7\" (UniqueName: \"kubernetes.io/projected/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-kube-api-access-bj8q7\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.352274 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.351429    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a21c0317-3ef0-4b2d-ae52-809097383928-multus-daemon-config\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.352274 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.351458    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6kqx\" (UniqueName: \"kubernetes.io/projected/a68be042-a345-44a5-9564-1d6d16d11b8a-kube-api-access-v6kqx\") pod \"node-ca-x75tt\" (UID: \"a68be042-a345-44a5-9564-1d6d16d11b8a\") " pod="openshift-image-registry/node-ca-x75tt"
Sep 06 19:11:08.352274 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.351485    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-etc-modprobe-d\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.352274 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.351507    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-etc-sysctl-conf\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.352274 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.351618    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b554b5b-8ce8-4674-b7d3-235a7c0b777b-system-cni-dir\") pod \"multus-additional-cni-plugins-q4ncl\" (UID: \"0b554b5b-8ce8-4674-b7d3-235a7c0b777b\") " pod="openshift-multus/multus-additional-cni-plugins-q4ncl"
Sep 06 19:11:08.352274 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.351679    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4557de3f-1c4f-448f-aec4-4552db157e3e-sys-fs\") pod \"aws-ebs-csi-driver-node-r4ggs\" (UID: \"4557de3f-1c4f-448f-aec4-4552db157e3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs"
Sep 06 19:11:08.352274 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.351778    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4557de3f-1c4f-448f-aec4-4552db157e3e-device-dir\") pod \"aws-ebs-csi-driver-node-r4ggs\" (UID: \"4557de3f-1c4f-448f-aec4-4552db157e3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs"
Sep 06 19:11:08.352699 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.352455    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4557de3f-1c4f-448f-aec4-4552db157e3e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r4ggs\" (UID: \"4557de3f-1c4f-448f-aec4-4552db157e3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs"
Sep 06 19:11:08.352699 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.351534    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-host-run-k8s-cni-cncf-io\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.352699 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.352511    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpqkl\" (UniqueName: \"kubernetes.io/projected/a21c0317-3ef0-4b2d-ae52-809097383928-kube-api-access-cpqkl\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.352699 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.352542    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-host-run-ovn-kubernetes\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.352699 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.352571    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9254e6f1-b73d-49cc-8bc1-513925ccde4f-env-overrides\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.352699 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.352628    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 06 19:11:08.352699 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.352640    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/30fe2339-bf82-4be5-8cf4-f68f98014b4f-konnectivity-ca\") pod \"konnectivity-agent-b5vzh\" (UID: \"30fe2339-bf82-4be5-8cf4-f68f98014b4f\") " pod="kube-system/konnectivity-agent-b5vzh"
Sep 06 19:11:08.353199 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.353177    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/30fe2339-bf82-4be5-8cf4-f68f98014b4f-konnectivity-ca\") pod \"konnectivity-agent-b5vzh\" (UID: \"30fe2339-bf82-4be5-8cf4-f68f98014b4f\") " pod="kube-system/konnectivity-agent-b5vzh"
Sep 06 19:11:08.359957 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.359929    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/30fe2339-bf82-4be5-8cf4-f68f98014b4f-agent-certs\") pod \"konnectivity-agent-b5vzh\" (UID: \"30fe2339-bf82-4be5-8cf4-f68f98014b4f\") " pod="kube-system/konnectivity-agent-b5vzh"
Sep 06 19:11:08.362404 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.362330    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df58r\" (UniqueName: \"kubernetes.io/projected/176abf0b-9b68-46d3-a657-ab219939488a-kube-api-access-df58r\") pod \"node-resolver-jm5gt\" (UID: \"176abf0b-9b68-46d3-a657-ab219939488a\") " pod="openshift-dns/node-resolver-jm5gt"
Sep 06 19:11:08.363819 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.363739    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhgf7\" (UniqueName: \"kubernetes.io/projected/4557de3f-1c4f-448f-aec4-4552db157e3e-kube-api-access-mhgf7\") pod \"aws-ebs-csi-driver-node-r4ggs\" (UID: \"4557de3f-1c4f-448f-aec4-4552db157e3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs"
Sep 06 19:11:08.365966 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.365926    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24wl8\" (UniqueName: \"kubernetes.io/projected/0b554b5b-8ce8-4674-b7d3-235a7c0b777b-kube-api-access-24wl8\") pod \"multus-additional-cni-plugins-q4ncl\" (UID: \"0b554b5b-8ce8-4674-b7d3-235a7c0b777b\") " pod="openshift-multus/multus-additional-cni-plugins-q4ncl"
Sep 06 19:11:08.453691 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.453621    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a68be042-a345-44a5-9564-1d6d16d11b8a-serviceca\") pod \"node-ca-x75tt\" (UID: \"a68be042-a345-44a5-9564-1d6d16d11b8a\") " pod="openshift-image-registry/node-ca-x75tt"
Sep 06 19:11:08.453691 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.453653    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.453691 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.453677    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-multus-cni-dir\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.453691 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.453693    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-host-run-netns\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.453927 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.453708    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-run-openvswitch\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.453927 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.453724    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-run-systemd\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.453927 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.453737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9254e6f1-b73d-49cc-8bc1-513925ccde4f-ovnkube-config\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.453927 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.453768    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99gc7\" (UniqueName: \"kubernetes.io/projected/9254e6f1-b73d-49cc-8bc1-513925ccde4f-kube-api-access-99gc7\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.453927 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.453802    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-host-var-lib-kubelet\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.453927 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.453816    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a68be042-a345-44a5-9564-1d6d16d11b8a-host\") pod \"node-ca-x75tt\" (UID: \"a68be042-a345-44a5-9564-1d6d16d11b8a\") " pod="openshift-image-registry/node-ca-x75tt"
Sep 06 19:11:08.453927 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.453829    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-node-log\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.453927 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.453841    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-log-socket\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.453927 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.453855    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9254e6f1-b73d-49cc-8bc1-513925ccde4f-ovn-node-metrics-cert\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.453927 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.453870    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-etc-sysctl-d\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.453927 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.453883    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-lib-modules\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.453927 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.453911    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-host-var-lib-cni-multus\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.453927 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.453925    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65fng\" (UniqueName: \"kubernetes.io/projected/83a7379b-ab58-4439-998e-448e48150f3b-kube-api-access-65fng\") pod \"network-metrics-daemon-qlvsw\" (UID: \"83a7379b-ab58-4439-998e-448e48150f3b\") " pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:08.454390 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.453939    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-etc-tuned\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.454390 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.453952    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9254e6f1-b73d-49cc-8bc1-513925ccde4f-ovnkube-script-lib\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.454390 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.453966    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs\") pod \"network-metrics-daemon-qlvsw\" (UID: \"83a7379b-ab58-4439-998e-448e48150f3b\") " pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:08.454390 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.453979    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-sys\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.454390 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.453994    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/317e1dba-5d84-4cfa-84aa-7d1a3dc399d5-iptables-alerter-script\") pod \"iptables-alerter-jdfqm\" (UID: \"317e1dba-5d84-4cfa-84aa-7d1a3dc399d5\") " pod="openshift-network-operator/iptables-alerter-jdfqm"
Sep 06 19:11:08.454390 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454008    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-host-cni-netd\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.454390 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454026    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-system-cni-dir\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.454390 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454047    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-multus-socket-dir-parent\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.454390 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454077    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-host-var-lib-cni-bin\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.454390 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454093    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jllc8\" (UniqueName: \"kubernetes.io/projected/317e1dba-5d84-4cfa-84aa-7d1a3dc399d5-kube-api-access-jllc8\") pod \"iptables-alerter-jdfqm\" (UID: \"317e1dba-5d84-4cfa-84aa-7d1a3dc399d5\") " pod="openshift-network-operator/iptables-alerter-jdfqm"
Sep 06 19:11:08.454390 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454108    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-var-lib-kubelet\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.454390 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454122    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-multus-conf-dir\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.454390 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454144    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-host-run-multus-certs\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.454390 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454157    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-run\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.454390 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454187    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-tmp\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.454390 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454202    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-var-lib-openvswitch\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.454390 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454215    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-etc-openvswitch\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.455007 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454244    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-etc-systemd\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.455007 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454258    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a21c0317-3ef0-4b2d-ae52-809097383928-cni-binary-copy\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.455007 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454279    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-host-cni-bin\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.455007 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454308    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-host\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.455007 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454322    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-cnibin\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.455007 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454336    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-hostroot\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.455007 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454351    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kvr2\" (UniqueName: \"kubernetes.io/projected/52bd7358-7476-41f4-95da-5bdfefff01db-kube-api-access-5kvr2\") pod \"network-check-target-s9h5h\" (UID: \"52bd7358-7476-41f4-95da-5bdfefff01db\") " pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:08.455007 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454365    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/317e1dba-5d84-4cfa-84aa-7d1a3dc399d5-host-slash\") pod \"iptables-alerter-jdfqm\" (UID: \"317e1dba-5d84-4cfa-84aa-7d1a3dc399d5\") " pod="openshift-network-operator/iptables-alerter-jdfqm"
Sep 06 19:11:08.455007 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454384    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-etc-sysconfig\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.455007 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454399    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-host-kubelet\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.455007 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454414    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-systemd-units\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.455007 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454429    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-host-slash\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.455007 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454444    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-run-ovn\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.455007 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454460    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-etc-kubernetes\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.455007 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454474    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-etc-kubernetes\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.455007 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454498    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-host-run-netns\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.455007 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454518    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj8q7\" (UniqueName: \"kubernetes.io/projected/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-kube-api-access-bj8q7\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.455007 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454537    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a21c0317-3ef0-4b2d-ae52-809097383928-multus-daemon-config\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.455635 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454552    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6kqx\" (UniqueName: \"kubernetes.io/projected/a68be042-a345-44a5-9564-1d6d16d11b8a-kube-api-access-v6kqx\") pod \"node-ca-x75tt\" (UID: \"a68be042-a345-44a5-9564-1d6d16d11b8a\") " pod="openshift-image-registry/node-ca-x75tt"
Sep 06 19:11:08.455635 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454571    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-etc-modprobe-d\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.455635 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454585    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-etc-sysctl-conf\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.455635 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454615    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-host-run-k8s-cni-cncf-io\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.455635 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454648    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpqkl\" (UniqueName: \"kubernetes.io/projected/a21c0317-3ef0-4b2d-ae52-809097383928-kube-api-access-cpqkl\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.455635 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454663    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-host-run-ovn-kubernetes\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.455635 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.454689    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9254e6f1-b73d-49cc-8bc1-513925ccde4f-env-overrides\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.455635 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.455159    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9254e6f1-b73d-49cc-8bc1-513925ccde4f-env-overrides\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.455635 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.455474    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a68be042-a345-44a5-9564-1d6d16d11b8a-serviceca\") pod \"node-ca-x75tt\" (UID: \"a68be042-a345-44a5-9564-1d6d16d11b8a\") " pod="openshift-image-registry/node-ca-x75tt"
Sep 06 19:11:08.455635 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.455517    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.455635 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.455564    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-multus-cni-dir\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.455635 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.455597    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-host-run-netns\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.456140 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.455658    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-run-openvswitch\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.456140 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.455684    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-run-systemd\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.456140 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.456024    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9254e6f1-b73d-49cc-8bc1-513925ccde4f-ovnkube-config\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.456263 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.456241    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-host-var-lib-kubelet\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.456322 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.456277    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a68be042-a345-44a5-9564-1d6d16d11b8a-host\") pod \"node-ca-x75tt\" (UID: \"a68be042-a345-44a5-9564-1d6d16d11b8a\") " pod="openshift-image-registry/node-ca-x75tt"
Sep 06 19:11:08.456322 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.456317    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-node-log\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.456402 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.456355    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-log-socket\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.458115 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.456761    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-host-run-multus-certs\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.458115 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.456799    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-system-cni-dir\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.458115 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.456798    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-run-ovn\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.458115 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.456836    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-multus-socket-dir-parent\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.458115 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.456838    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-host-cni-bin\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.458115 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.456881    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-host\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.458115 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.456891    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-host-var-lib-cni-bin\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.458115 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457072    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/317e1dba-5d84-4cfa-84aa-7d1a3dc399d5-host-slash\") pod \"iptables-alerter-jdfqm\" (UID: \"317e1dba-5d84-4cfa-84aa-7d1a3dc399d5\") " pod="openshift-network-operator/iptables-alerter-jdfqm"
Sep 06 19:11:08.458115 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457130    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-etc-sysconfig\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.458115 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457150    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9254e6f1-b73d-49cc-8bc1-513925ccde4f-ovnkube-script-lib\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.458115 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457156    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a21c0317-3ef0-4b2d-ae52-809097383928-cni-binary-copy\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.458115 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457171    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-host-kubelet\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.458115 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457206    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-multus-conf-dir\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.458115 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:08.457211    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:11:08.458115 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457222    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-run\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.458115 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457230    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-cnibin\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.458115 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457248    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-systemd-units\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.458115 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:08.457259    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs podName:83a7379b-ab58-4439-998e-448e48150f3b nodeName:}" failed. No retries permitted until 2026-09-06 19:11:08.957246108 +0000 UTC m=+3.138427892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs") pod "network-metrics-daemon-qlvsw" (UID: "83a7379b-ab58-4439-998e-448e48150f3b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:11:08.459292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457257    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-hostroot\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.459292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457169    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-var-lib-kubelet\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.459292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457281    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-etc-kubernetes\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.459292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457306    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-sys\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.459292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457310    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-var-lib-openvswitch\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.459292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457317    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-etc-openvswitch\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.459292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457330    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-etc-kubernetes\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.459292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457341    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-host-slash\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.459292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457376    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-etc-systemd\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.459292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457388    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-host-cni-netd\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.459292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457392    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-host-run-netns\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.459292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457639    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-host-run-k8s-cni-cncf-io\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.459292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457656    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9254e6f1-b73d-49cc-8bc1-513925ccde4f-host-run-ovn-kubernetes\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.459292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457757    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-etc-modprobe-d\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.459292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457798    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-etc-sysctl-conf\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.459292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457833    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a21c0317-3ef0-4b2d-ae52-809097383928-multus-daemon-config\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.459292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457878    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-lib-modules\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.459292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457917    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-etc-sysctl-d\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.460212 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.457940    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a21c0317-3ef0-4b2d-ae52-809097383928-host-var-lib-cni-multus\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.460212 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.458391    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/317e1dba-5d84-4cfa-84aa-7d1a3dc399d5-iptables-alerter-script\") pod \"iptables-alerter-jdfqm\" (UID: \"317e1dba-5d84-4cfa-84aa-7d1a3dc399d5\") " pod="openshift-network-operator/iptables-alerter-jdfqm"
Sep 06 19:11:08.460742 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.460503    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-tmp\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.460742 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.460705    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9254e6f1-b73d-49cc-8bc1-513925ccde4f-ovn-node-metrics-cert\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.460840 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.460770    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-etc-tuned\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.463800 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:08.463572    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 19:11:08.463800 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:08.463593    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 19:11:08.463800 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:08.463623    2566 projected.go:196] Error preparing data for projected volume kube-api-access-5kvr2 for pod openshift-network-diagnostics/network-check-target-s9h5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:11:08.463800 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:08.463690    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52bd7358-7476-41f4-95da-5bdfefff01db-kube-api-access-5kvr2 podName:52bd7358-7476-41f4-95da-5bdfefff01db nodeName:}" failed. No retries permitted until 2026-09-06 19:11:08.963674006 +0000 UTC m=+3.144855796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5kvr2" (UniqueName: "kubernetes.io/projected/52bd7358-7476-41f4-95da-5bdfefff01db-kube-api-access-5kvr2") pod "network-check-target-s9h5h" (UID: "52bd7358-7476-41f4-95da-5bdfefff01db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:11:08.466025 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.465922    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jllc8\" (UniqueName: \"kubernetes.io/projected/317e1dba-5d84-4cfa-84aa-7d1a3dc399d5-kube-api-access-jllc8\") pod \"iptables-alerter-jdfqm\" (UID: \"317e1dba-5d84-4cfa-84aa-7d1a3dc399d5\") " pod="openshift-network-operator/iptables-alerter-jdfqm"
Sep 06 19:11:08.467235 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.467182    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99gc7\" (UniqueName: \"kubernetes.io/projected/9254e6f1-b73d-49cc-8bc1-513925ccde4f-kube-api-access-99gc7\") pod \"ovnkube-node-gnssq\" (UID: \"9254e6f1-b73d-49cc-8bc1-513925ccde4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.468782 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.468747    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpqkl\" (UniqueName: \"kubernetes.io/projected/a21c0317-3ef0-4b2d-ae52-809097383928-kube-api-access-cpqkl\") pod \"multus-tb692\" (UID: \"a21c0317-3ef0-4b2d-ae52-809097383928\") " pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.469438 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.469392    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6kqx\" (UniqueName: \"kubernetes.io/projected/a68be042-a345-44a5-9564-1d6d16d11b8a-kube-api-access-v6kqx\") pod \"node-ca-x75tt\" (UID: \"a68be042-a345-44a5-9564-1d6d16d11b8a\") " pod="openshift-image-registry/node-ca-x75tt"
Sep 06 19:11:08.469615 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.469581    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65fng\" (UniqueName: \"kubernetes.io/projected/83a7379b-ab58-4439-998e-448e48150f3b-kube-api-access-65fng\") pod \"network-metrics-daemon-qlvsw\" (UID: \"83a7379b-ab58-4439-998e-448e48150f3b\") " pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:08.471417 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.471396    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj8q7\" (UniqueName: \"kubernetes.io/projected/5c08a764-6c0b-48c6-a532-9d0b0b00ccb1-kube-api-access-bj8q7\") pod \"tuned-8hj5v\" (UID: \"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1\") " pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.546621 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.546577    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b5vzh"
Sep 06 19:11:08.557549 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.557530    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q4ncl"
Sep 06 19:11:08.566263 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.566247    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs"
Sep 06 19:11:08.568774 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.568751    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 19:11:08.571549 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.571530    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jm5gt"
Sep 06 19:11:08.580234 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.580215    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x75tt"
Sep 06 19:11:08.589194 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.589175    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jdfqm"
Sep 06 19:11:08.598954 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.598924    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tb692"
Sep 06 19:11:08.606754 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.606590    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:08.613336 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.613321    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8hj5v"
Sep 06 19:11:08.683182 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:08.683146    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b554b5b_8ce8_4674_b7d3_235a7c0b777b.slice/crio-9bc7dd82f475cc66398ba0e7496858809d6b613b7054c6e3e89400294d011c64 WatchSource:0}: Error finding container 9bc7dd82f475cc66398ba0e7496858809d6b613b7054c6e3e89400294d011c64: Status 404 returned error can't find the container with id 9bc7dd82f475cc66398ba0e7496858809d6b613b7054c6e3e89400294d011c64
Sep 06 19:11:08.686701 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:08.686670    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30fe2339_bf82_4be5_8cf4_f68f98014b4f.slice/crio-431b88951268665730112063c8fbc5af2261c0b01f4420c53d7d26f45c86980a WatchSource:0}: Error finding container 431b88951268665730112063c8fbc5af2261c0b01f4420c53d7d26f45c86980a: Status 404 returned error can't find the container with id 431b88951268665730112063c8fbc5af2261c0b01f4420c53d7d26f45c86980a
Sep 06 19:11:08.688079 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:08.688056    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda68be042_a345_44a5_9564_1d6d16d11b8a.slice/crio-50b014969972854aabad50556a9fc7ee1a2610331558ce11c04c0396e1dd2a05 WatchSource:0}: Error finding container 50b014969972854aabad50556a9fc7ee1a2610331558ce11c04c0396e1dd2a05: Status 404 returned error can't find the container with id 50b014969972854aabad50556a9fc7ee1a2610331558ce11c04c0396e1dd2a05
Sep 06 19:11:08.689725 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:08.689698    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9254e6f1_b73d_49cc_8bc1_513925ccde4f.slice/crio-9d9484ef7a3fde45ae63534e85c3b1559c5797774f2ead8a5e79afcd6ad3f32c WatchSource:0}: Error finding container 9d9484ef7a3fde45ae63534e85c3b1559c5797774f2ead8a5e79afcd6ad3f32c: Status 404 returned error can't find the container with id 9d9484ef7a3fde45ae63534e85c3b1559c5797774f2ead8a5e79afcd6ad3f32c
Sep 06 19:11:08.690996 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:08.690724    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176abf0b_9b68_46d3_a657_ab219939488a.slice/crio-2d10c9b9c82d17071d8d26d1135f85b3d8b793bc6bcfc3b057732fa539d0a112 WatchSource:0}: Error finding container 2d10c9b9c82d17071d8d26d1135f85b3d8b793bc6bcfc3b057732fa539d0a112: Status 404 returned error can't find the container with id 2d10c9b9c82d17071d8d26d1135f85b3d8b793bc6bcfc3b057732fa539d0a112
Sep 06 19:11:08.691279 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:08.691261    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c08a764_6c0b_48c6_a532_9d0b0b00ccb1.slice/crio-184bc5b174a167ddb63f62739000fa879b8c3dc4720e375988f7b95eea8ff81f WatchSource:0}: Error finding container 184bc5b174a167ddb63f62739000fa879b8c3dc4720e375988f7b95eea8ff81f: Status 404 returned error can't find the container with id 184bc5b174a167ddb63f62739000fa879b8c3dc4720e375988f7b95eea8ff81f
Sep 06 19:11:08.693937 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:08.693917    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317e1dba_5d84_4cfa_84aa_7d1a3dc399d5.slice/crio-9446dffa6e8e5f576be902f119f63f040309b8f887f9390516f3f71a138073f3 WatchSource:0}: Error finding container 9446dffa6e8e5f576be902f119f63f040309b8f887f9390516f3f71a138073f3: Status 404 returned error can't find the container with id 9446dffa6e8e5f576be902f119f63f040309b8f887f9390516f3f71a138073f3
Sep 06 19:11:08.695241 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:08.695102    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21c0317_3ef0_4b2d_ae52_809097383928.slice/crio-97109386c7bc09d469d707d0c9ed9655466ee1295579daac99125c44bffd8586 WatchSource:0}: Error finding container 97109386c7bc09d469d707d0c9ed9655466ee1295579daac99125c44bffd8586: Status 404 returned error can't find the container with id 97109386c7bc09d469d707d0c9ed9655466ee1295579daac99125c44bffd8586
Sep 06 19:11:08.857481 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.857439    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-original-pull-secret\") pod \"global-pull-secret-syncer-7crp9\" (UID: \"dcd6e3bb-3d92-4fe9-8432-62ef628b2472\") " pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:08.857656 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:08.857593    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 19:11:08.857723 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:08.857672    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-original-pull-secret podName:dcd6e3bb-3d92-4fe9-8432-62ef628b2472 nodeName:}" failed. No retries permitted until 2026-09-06 19:11:09.857652366 +0000 UTC m=+4.038834166 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-original-pull-secret") pod "global-pull-secret-syncer-7crp9" (UID: "dcd6e3bb-3d92-4fe9-8432-62ef628b2472") : object "kube-system"/"original-pull-secret" not registered
Sep 06 19:11:08.958586 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:08.958553    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs\") pod \"network-metrics-daemon-qlvsw\" (UID: \"83a7379b-ab58-4439-998e-448e48150f3b\") " pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:08.958743 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:08.958694    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:11:08.958781 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:08.958745    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs podName:83a7379b-ab58-4439-998e-448e48150f3b nodeName:}" failed. No retries permitted until 2026-09-06 19:11:09.958731962 +0000 UTC m=+4.139913745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs") pod "network-metrics-daemon-qlvsw" (UID: "83a7379b-ab58-4439-998e-448e48150f3b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:11:09.041663 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:09.041576    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4557de3f_1c4f_448f_aec4_4552db157e3e.slice/crio-3db4fe789b56cf3abfd0584b91e2c879338cd9afc306875f26b16bc64da4056a WatchSource:0}: Error finding container 3db4fe789b56cf3abfd0584b91e2c879338cd9afc306875f26b16bc64da4056a: Status 404 returned error can't find the container with id 3db4fe789b56cf3abfd0584b91e2c879338cd9afc306875f26b16bc64da4056a
Sep 06 19:11:09.059077 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:09.059053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kvr2\" (UniqueName: \"kubernetes.io/projected/52bd7358-7476-41f4-95da-5bdfefff01db-kube-api-access-5kvr2\") pod \"network-check-target-s9h5h\" (UID: \"52bd7358-7476-41f4-95da-5bdfefff01db\") " pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:09.059226 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:09.059207    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 19:11:09.059290 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:09.059232    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 19:11:09.059290 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:09.059245    2566 projected.go:196] Error preparing data for projected volume kube-api-access-5kvr2 for pod openshift-network-diagnostics/network-check-target-s9h5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:11:09.059387 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:09.059306    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52bd7358-7476-41f4-95da-5bdfefff01db-kube-api-access-5kvr2 podName:52bd7358-7476-41f4-95da-5bdfefff01db nodeName:}" failed. No retries permitted until 2026-09-06 19:11:10.059285349 +0000 UTC m=+4.240467141 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5kvr2" (UniqueName: "kubernetes.io/projected/52bd7358-7476-41f4-95da-5bdfefff01db-kube-api-access-5kvr2") pod "network-check-target-s9h5h" (UID: "52bd7358-7476-41f4-95da-5bdfefff01db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:11:09.280029 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:09.279968    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 19:06:08 +0000 UTC" deadline="2028-02-29 07:27:46.349248585 +0000 UTC"
Sep 06 19:11:09.280029 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:09.280026    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12972h16m37.069227091s"
Sep 06 19:11:09.316947 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:09.316866    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tb692" event={"ID":"a21c0317-3ef0-4b2d-ae52-809097383928","Type":"ContainerStarted","Data":"97109386c7bc09d469d707d0c9ed9655466ee1295579daac99125c44bffd8586"}
Sep 06 19:11:09.318660 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:09.318628    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8hj5v" event={"ID":"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1","Type":"ContainerStarted","Data":"184bc5b174a167ddb63f62739000fa879b8c3dc4720e375988f7b95eea8ff81f"}
Sep 06 19:11:09.320251 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:09.320190    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jm5gt" event={"ID":"176abf0b-9b68-46d3-a657-ab219939488a","Type":"ContainerStarted","Data":"2d10c9b9c82d17071d8d26d1135f85b3d8b793bc6bcfc3b057732fa539d0a112"}
Sep 06 19:11:09.321626 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:09.321587    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b5vzh" event={"ID":"30fe2339-bf82-4be5-8cf4-f68f98014b4f","Type":"ContainerStarted","Data":"431b88951268665730112063c8fbc5af2261c0b01f4420c53d7d26f45c86980a"}
Sep 06 19:11:09.323065 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:09.323037    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jdfqm" event={"ID":"317e1dba-5d84-4cfa-84aa-7d1a3dc399d5","Type":"ContainerStarted","Data":"9446dffa6e8e5f576be902f119f63f040309b8f887f9390516f3f71a138073f3"}
Sep 06 19:11:09.324462 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:09.324439    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnssq" event={"ID":"9254e6f1-b73d-49cc-8bc1-513925ccde4f","Type":"ContainerStarted","Data":"9d9484ef7a3fde45ae63534e85c3b1559c5797774f2ead8a5e79afcd6ad3f32c"}
Sep 06 19:11:09.325763 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:09.325737    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x75tt" event={"ID":"a68be042-a345-44a5-9564-1d6d16d11b8a","Type":"ContainerStarted","Data":"50b014969972854aabad50556a9fc7ee1a2610331558ce11c04c0396e1dd2a05"}
Sep 06 19:11:09.326988 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:09.326964    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4ncl" event={"ID":"0b554b5b-8ce8-4674-b7d3-235a7c0b777b","Type":"ContainerStarted","Data":"9bc7dd82f475cc66398ba0e7496858809d6b613b7054c6e3e89400294d011c64"}
Sep 06 19:11:09.328209 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:09.328183    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs" event={"ID":"4557de3f-1c4f-448f-aec4-4552db157e3e","Type":"ContainerStarted","Data":"3db4fe789b56cf3abfd0584b91e2c879338cd9afc306875f26b16bc64da4056a"}
Sep 06 19:11:09.865088 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:09.865053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-original-pull-secret\") pod \"global-pull-secret-syncer-7crp9\" (UID: \"dcd6e3bb-3d92-4fe9-8432-62ef628b2472\") " pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:09.865251 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:09.865241    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 19:11:09.865328 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:09.865298    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-original-pull-secret podName:dcd6e3bb-3d92-4fe9-8432-62ef628b2472 nodeName:}" failed. No retries permitted until 2026-09-06 19:11:11.865280615 +0000 UTC m=+6.046462404 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-original-pull-secret") pod "global-pull-secret-syncer-7crp9" (UID: "dcd6e3bb-3d92-4fe9-8432-62ef628b2472") : object "kube-system"/"original-pull-secret" not registered
Sep 06 19:11:09.967030 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:09.966528    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs\") pod \"network-metrics-daemon-qlvsw\" (UID: \"83a7379b-ab58-4439-998e-448e48150f3b\") " pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:09.967030 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:09.966647    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:11:09.967030 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:09.966705    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs podName:83a7379b-ab58-4439-998e-448e48150f3b nodeName:}" failed. No retries permitted until 2026-09-06 19:11:11.966685717 +0000 UTC m=+6.147867501 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs") pod "network-metrics-daemon-qlvsw" (UID: "83a7379b-ab58-4439-998e-448e48150f3b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:11:10.067058 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:10.067000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kvr2\" (UniqueName: \"kubernetes.io/projected/52bd7358-7476-41f4-95da-5bdfefff01db-kube-api-access-5kvr2\") pod \"network-check-target-s9h5h\" (UID: \"52bd7358-7476-41f4-95da-5bdfefff01db\") " pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:10.067216 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:10.067202    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 19:11:10.067281 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:10.067222    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 19:11:10.067281 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:10.067235    2566 projected.go:196] Error preparing data for projected volume kube-api-access-5kvr2 for pod openshift-network-diagnostics/network-check-target-s9h5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:11:10.067376 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:10.067289    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52bd7358-7476-41f4-95da-5bdfefff01db-kube-api-access-5kvr2 podName:52bd7358-7476-41f4-95da-5bdfefff01db nodeName:}" failed. No retries permitted until 2026-09-06 19:11:12.067273371 +0000 UTC m=+6.248455159 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5kvr2" (UniqueName: "kubernetes.io/projected/52bd7358-7476-41f4-95da-5bdfefff01db-kube-api-access-5kvr2") pod "network-check-target-s9h5h" (UID: "52bd7358-7476-41f4-95da-5bdfefff01db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:11:10.281103 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:10.281045    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 19:06:08 +0000 UTC" deadline="2028-05-07 22:23:31.704141028 +0000 UTC"
Sep 06 19:11:10.281103 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:10.281084    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14619h12m21.423061796s"
Sep 06 19:11:10.311295 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:10.311266    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:10.311446 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:10.311384    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9h5h" podUID="52bd7358-7476-41f4-95da-5bdfefff01db"
Sep 06 19:11:10.311820 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:10.311800    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:10.311924 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:10.311904    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7crp9" podUID="dcd6e3bb-3d92-4fe9-8432-62ef628b2472"
Sep 06 19:11:10.311989 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:10.311961    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:10.312095 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:10.312072    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlvsw" podUID="83a7379b-ab58-4439-998e-448e48150f3b"
Sep 06 19:11:10.343731 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:10.343701    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.ec2.internal" event={"ID":"faae1a37c3a006a83d2995caac1d3848","Type":"ContainerStarted","Data":"6a1be1b0a8334d678861357863835a67cab4d862ef6269c1b22b12409ecdec3c"}
Sep 06 19:11:10.349134 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:10.348589    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.ec2.internal" event={"ID":"1a6b312538c38d2cf66321b95dbc03a9","Type":"ContainerStarted","Data":"402864f27484836fe84a359a020377a085d17134e3a4acea2ac049fcb2d45c03"}
Sep 06 19:11:10.373002 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:10.372049    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.ec2.internal" podStartSLOduration=3.37203081 podStartE2EDuration="3.37203081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:11:10.357806976 +0000 UTC m=+4.538988782" watchObservedRunningTime="2026-09-06 19:11:10.37203081 +0000 UTC m=+4.553212618"
Sep 06 19:11:11.882416 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:11.882385    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-original-pull-secret\") pod \"global-pull-secret-syncer-7crp9\" (UID: \"dcd6e3bb-3d92-4fe9-8432-62ef628b2472\") " pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:11.882843 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:11.882558    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 19:11:11.882843 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:11.882629    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-original-pull-secret podName:dcd6e3bb-3d92-4fe9-8432-62ef628b2472 nodeName:}" failed. No retries permitted until 2026-09-06 19:11:15.882596062 +0000 UTC m=+10.063777851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-original-pull-secret") pod "global-pull-secret-syncer-7crp9" (UID: "dcd6e3bb-3d92-4fe9-8432-62ef628b2472") : object "kube-system"/"original-pull-secret" not registered
Sep 06 19:11:11.983644 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:11.983006    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs\") pod \"network-metrics-daemon-qlvsw\" (UID: \"83a7379b-ab58-4439-998e-448e48150f3b\") " pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:11.983644 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:11.983187    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:11:11.983644 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:11.983246    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs podName:83a7379b-ab58-4439-998e-448e48150f3b nodeName:}" failed. No retries permitted until 2026-09-06 19:11:15.983226355 +0000 UTC m=+10.164408158 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs") pod "network-metrics-daemon-qlvsw" (UID: "83a7379b-ab58-4439-998e-448e48150f3b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:11:12.086672 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:12.083808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kvr2\" (UniqueName: \"kubernetes.io/projected/52bd7358-7476-41f4-95da-5bdfefff01db-kube-api-access-5kvr2\") pod \"network-check-target-s9h5h\" (UID: \"52bd7358-7476-41f4-95da-5bdfefff01db\") " pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:12.090631 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:12.088416    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 19:11:12.090631 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:12.088444    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 19:11:12.090631 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:12.088460    2566 projected.go:196] Error preparing data for projected volume kube-api-access-5kvr2 for pod openshift-network-diagnostics/network-check-target-s9h5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:11:12.090631 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:12.088524    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52bd7358-7476-41f4-95da-5bdfefff01db-kube-api-access-5kvr2 podName:52bd7358-7476-41f4-95da-5bdfefff01db nodeName:}" failed. No retries permitted until 2026-09-06 19:11:16.088504035 +0000 UTC m=+10.269685821 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5kvr2" (UniqueName: "kubernetes.io/projected/52bd7358-7476-41f4-95da-5bdfefff01db-kube-api-access-5kvr2") pod "network-check-target-s9h5h" (UID: "52bd7358-7476-41f4-95da-5bdfefff01db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:11:12.304474 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:12.304440    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:12.304668 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:12.304576    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlvsw" podUID="83a7379b-ab58-4439-998e-448e48150f3b"
Sep 06 19:11:12.304729 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:12.304673    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:12.304777 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:12.304743    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9h5h" podUID="52bd7358-7476-41f4-95da-5bdfefff01db"
Sep 06 19:11:12.304811 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:12.304799    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:12.304881 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:12.304863    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7crp9" podUID="dcd6e3bb-3d92-4fe9-8432-62ef628b2472"
Sep 06 19:11:12.355983 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:12.355149    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b5vzh" event={"ID":"30fe2339-bf82-4be5-8cf4-f68f98014b4f","Type":"ContainerStarted","Data":"583fcd1808a8b4094623bc986bfb6d14392081b1ae182061e48756111923b797"}
Sep 06 19:11:12.359583 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:12.358977    2566 generic.go:356] "Generic (PLEG): container finished" podID="1a6b312538c38d2cf66321b95dbc03a9" containerID="402864f27484836fe84a359a020377a085d17134e3a4acea2ac049fcb2d45c03" exitCode=0
Sep 06 19:11:12.359583 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:12.359024    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.ec2.internal" event={"ID":"1a6b312538c38d2cf66321b95dbc03a9","Type":"ContainerDied","Data":"402864f27484836fe84a359a020377a085d17134e3a4acea2ac049fcb2d45c03"}
Sep 06 19:11:12.413982 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:12.413926    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b5vzh" podStartSLOduration=3.487105691 podStartE2EDuration="6.413905624s" totalImagesPullingTime="2.926799933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:11:12.388739818 +0000 UTC m=+6.569921630" watchObservedRunningTime="2026-09-06 19:11:12.413905624 +0000 UTC m=+6.595087431"
Sep 06 19:11:13.469819 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:13.469788    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b5vzh"
Sep 06 19:11:13.470291 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:13.470273    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b5vzh"
Sep 06 19:11:14.308727 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:14.308697    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:14.308879 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:14.308820    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7crp9" podUID="dcd6e3bb-3d92-4fe9-8432-62ef628b2472"
Sep 06 19:11:14.309191 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:14.309165    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:14.309320 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:14.309279    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlvsw" podUID="83a7379b-ab58-4439-998e-448e48150f3b"
Sep 06 19:11:14.309394 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:14.309330    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:14.309446 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:14.309396    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9h5h" podUID="52bd7358-7476-41f4-95da-5bdfefff01db"
Sep 06 19:11:15.365139 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:15.365067    2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 06 19:11:15.919250 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:15.919212    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-original-pull-secret\") pod \"global-pull-secret-syncer-7crp9\" (UID: \"dcd6e3bb-3d92-4fe9-8432-62ef628b2472\") " pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:15.919456 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:15.919401    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 19:11:15.919519 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:15.919468    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-original-pull-secret podName:dcd6e3bb-3d92-4fe9-8432-62ef628b2472 nodeName:}" failed. No retries permitted until 2026-09-06 19:11:23.919453864 +0000 UTC m=+18.100635652 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-original-pull-secret") pod "global-pull-secret-syncer-7crp9" (UID: "dcd6e3bb-3d92-4fe9-8432-62ef628b2472") : object "kube-system"/"original-pull-secret" not registered
Sep 06 19:11:16.020186 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:16.020145    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs\") pod \"network-metrics-daemon-qlvsw\" (UID: \"83a7379b-ab58-4439-998e-448e48150f3b\") " pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:16.020355 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:16.020282    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:11:16.020355 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:16.020344    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs podName:83a7379b-ab58-4439-998e-448e48150f3b nodeName:}" failed. No retries permitted until 2026-09-06 19:11:24.020326031 +0000 UTC m=+18.201507817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs") pod "network-metrics-daemon-qlvsw" (UID: "83a7379b-ab58-4439-998e-448e48150f3b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:11:16.121252 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:16.121216    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kvr2\" (UniqueName: \"kubernetes.io/projected/52bd7358-7476-41f4-95da-5bdfefff01db-kube-api-access-5kvr2\") pod \"network-check-target-s9h5h\" (UID: \"52bd7358-7476-41f4-95da-5bdfefff01db\") " pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:16.121414 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:16.121342    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 19:11:16.121414 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:16.121357    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 19:11:16.121414 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:16.121366    2566 projected.go:196] Error preparing data for projected volume kube-api-access-5kvr2 for pod openshift-network-diagnostics/network-check-target-s9h5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:11:16.121414 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:16.121408    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52bd7358-7476-41f4-95da-5bdfefff01db-kube-api-access-5kvr2 podName:52bd7358-7476-41f4-95da-5bdfefff01db nodeName:}" failed. No retries permitted until 2026-09-06 19:11:24.121395748 +0000 UTC m=+18.302577546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5kvr2" (UniqueName: "kubernetes.io/projected/52bd7358-7476-41f4-95da-5bdfefff01db-kube-api-access-5kvr2") pod "network-check-target-s9h5h" (UID: "52bd7358-7476-41f4-95da-5bdfefff01db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:11:16.307650 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:16.307548    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:16.307650 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:16.307560    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:16.307860 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:16.307689    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7crp9" podUID="dcd6e3bb-3d92-4fe9-8432-62ef628b2472"
Sep 06 19:11:16.307860 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:16.307696    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:16.307860 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:16.307760    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlvsw" podUID="83a7379b-ab58-4439-998e-448e48150f3b"
Sep 06 19:11:16.307860 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:16.307840    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9h5h" podUID="52bd7358-7476-41f4-95da-5bdfefff01db"
Sep 06 19:11:18.304257 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:18.304222    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:18.304735 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:18.304230    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:18.304735 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:18.304340    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9h5h" podUID="52bd7358-7476-41f4-95da-5bdfefff01db"
Sep 06 19:11:18.304735 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:18.304239    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:18.304735 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:18.304408    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7crp9" podUID="dcd6e3bb-3d92-4fe9-8432-62ef628b2472"
Sep 06 19:11:18.304735 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:18.304484    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlvsw" podUID="83a7379b-ab58-4439-998e-448e48150f3b"
Sep 06 19:11:20.304312 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:20.304105    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:20.304746 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:20.304105    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:20.304746 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:20.304418    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlvsw" podUID="83a7379b-ab58-4439-998e-448e48150f3b"
Sep 06 19:11:20.304746 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:20.304137    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:20.304746 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:20.304499    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9h5h" podUID="52bd7358-7476-41f4-95da-5bdfefff01db"
Sep 06 19:11:20.304746 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:20.304572    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7crp9" podUID="dcd6e3bb-3d92-4fe9-8432-62ef628b2472"
Sep 06 19:11:22.304332 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:22.304142    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:22.304332 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:22.304173    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:22.304332 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:22.304144    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:22.304332 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:22.304254    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7crp9" podUID="dcd6e3bb-3d92-4fe9-8432-62ef628b2472"
Sep 06 19:11:22.304332 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:22.304312    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlvsw" podUID="83a7379b-ab58-4439-998e-448e48150f3b"
Sep 06 19:11:22.304992 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:22.304379    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9h5h" podUID="52bd7358-7476-41f4-95da-5bdfefff01db"
Sep 06 19:11:23.982077 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:23.982036    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-original-pull-secret\") pod \"global-pull-secret-syncer-7crp9\" (UID: \"dcd6e3bb-3d92-4fe9-8432-62ef628b2472\") " pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:23.982547 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:23.982232    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 19:11:23.982547 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:23.982306    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-original-pull-secret podName:dcd6e3bb-3d92-4fe9-8432-62ef628b2472 nodeName:}" failed. No retries permitted until 2026-09-06 19:11:39.982278975 +0000 UTC m=+34.163460759 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-original-pull-secret") pod "global-pull-secret-syncer-7crp9" (UID: "dcd6e3bb-3d92-4fe9-8432-62ef628b2472") : object "kube-system"/"original-pull-secret" not registered
Sep 06 19:11:24.082744 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:24.082703    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs\") pod \"network-metrics-daemon-qlvsw\" (UID: \"83a7379b-ab58-4439-998e-448e48150f3b\") " pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:24.082906 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:24.082874    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:11:24.082966 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:24.082954    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs podName:83a7379b-ab58-4439-998e-448e48150f3b nodeName:}" failed. No retries permitted until 2026-09-06 19:11:40.082929177 +0000 UTC m=+34.264110961 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs") pod "network-metrics-daemon-qlvsw" (UID: "83a7379b-ab58-4439-998e-448e48150f3b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:11:24.183879 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:24.183845    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kvr2\" (UniqueName: \"kubernetes.io/projected/52bd7358-7476-41f4-95da-5bdfefff01db-kube-api-access-5kvr2\") pod \"network-check-target-s9h5h\" (UID: \"52bd7358-7476-41f4-95da-5bdfefff01db\") " pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:24.184023 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:24.183993    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 19:11:24.184023 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:24.184010    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 19:11:24.184023 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:24.184023    2566 projected.go:196] Error preparing data for projected volume kube-api-access-5kvr2 for pod openshift-network-diagnostics/network-check-target-s9h5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:11:24.184167 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:24.184078    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52bd7358-7476-41f4-95da-5bdfefff01db-kube-api-access-5kvr2 podName:52bd7358-7476-41f4-95da-5bdfefff01db nodeName:}" failed. No retries permitted until 2026-09-06 19:11:40.184060798 +0000 UTC m=+34.365242587 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5kvr2" (UniqueName: "kubernetes.io/projected/52bd7358-7476-41f4-95da-5bdfefff01db-kube-api-access-5kvr2") pod "network-check-target-s9h5h" (UID: "52bd7358-7476-41f4-95da-5bdfefff01db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:11:24.304394 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:24.304304    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:24.304394 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:24.304324    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:24.304394 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:24.304339    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:24.304674 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:24.304446    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7crp9" podUID="dcd6e3bb-3d92-4fe9-8432-62ef628b2472"
Sep 06 19:11:24.304674 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:24.304499    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9h5h" podUID="52bd7358-7476-41f4-95da-5bdfefff01db"
Sep 06 19:11:24.304674 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:24.304555    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlvsw" podUID="83a7379b-ab58-4439-998e-448e48150f3b"
Sep 06 19:11:26.304853 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:26.304830    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:26.305130 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:26.304935    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9h5h" podUID="52bd7358-7476-41f4-95da-5bdfefff01db"
Sep 06 19:11:26.305277 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:26.305260    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:26.305377 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:26.305357    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7crp9" podUID="dcd6e3bb-3d92-4fe9-8432-62ef628b2472"
Sep 06 19:11:26.305434 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:26.305408    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:26.305504 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:26.305487    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlvsw" podUID="83a7379b-ab58-4439-998e-448e48150f3b"
Sep 06 19:11:27.389799 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:27.389582    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jm5gt" event={"ID":"176abf0b-9b68-46d3-a657-ab219939488a","Type":"ContainerStarted","Data":"c05e4b0a6aaff4ab58412a7299b81f72f082387ad6936406a346b0b03d4a7733"}
Sep 06 19:11:27.391860 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:27.391830    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.ec2.internal" event={"ID":"1a6b312538c38d2cf66321b95dbc03a9","Type":"ContainerStarted","Data":"3ed1cf0e9a474a78b803bf64d1fcfe70c2e5f8f284ea431825b55772b85489eb"}
Sep 06 19:11:27.395747 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:27.395727    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnssq_9254e6f1-b73d-49cc-8bc1-513925ccde4f/ovn-acl-logging/0.log"
Sep 06 19:11:27.396231 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:27.396204    2566 generic.go:356] "Generic (PLEG): container finished" podID="9254e6f1-b73d-49cc-8bc1-513925ccde4f" containerID="87d6432950ae1c16486ab31f6b3da50daf2543dff2a2187b2081dfd49748b795" exitCode=1
Sep 06 19:11:27.396336 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:27.396286    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnssq" event={"ID":"9254e6f1-b73d-49cc-8bc1-513925ccde4f","Type":"ContainerStarted","Data":"765747689fa4cd1afd832b5224db95bc0afddf7729d7dc3c58f42ebf090c880a"}
Sep 06 19:11:27.396401 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:27.396353    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnssq" event={"ID":"9254e6f1-b73d-49cc-8bc1-513925ccde4f","Type":"ContainerStarted","Data":"1c5b7f43a55c0d696011bd34a829c9e685c80b86ef8a35db84cefd29ff1e228b"}
Sep 06 19:11:27.396456 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:27.396405    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnssq" event={"ID":"9254e6f1-b73d-49cc-8bc1-513925ccde4f","Type":"ContainerStarted","Data":"089cf773b5b5b3c4ccd56f9d877d3851e4a4b7f160a6c8fb54b5267d127ef7fe"}
Sep 06 19:11:27.396505 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:27.396454    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnssq" event={"ID":"9254e6f1-b73d-49cc-8bc1-513925ccde4f","Type":"ContainerStarted","Data":"3a15574634c0bd47530c34132f7d9c37bead5f3a483566e7d511833243682c22"}
Sep 06 19:11:27.396505 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:27.396501    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnssq" event={"ID":"9254e6f1-b73d-49cc-8bc1-513925ccde4f","Type":"ContainerDied","Data":"87d6432950ae1c16486ab31f6b3da50daf2543dff2a2187b2081dfd49748b795"}
Sep 06 19:11:27.396630 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:27.396554    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnssq" event={"ID":"9254e6f1-b73d-49cc-8bc1-513925ccde4f","Type":"ContainerStarted","Data":"f3df381b6d26dd3cc133a4a524edf7598cf24eff4b642a22537bf9cd7db63cf0"}
Sep 06 19:11:27.397948 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:27.397927    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x75tt" event={"ID":"a68be042-a345-44a5-9564-1d6d16d11b8a","Type":"ContainerStarted","Data":"5c640c1bc0ccb9cc11262018ec95576b0e9b4cdf03d190416c27a5e2c55783f6"}
Sep 06 19:11:27.399592 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:27.399559    2566 generic.go:356] "Generic (PLEG): container finished" podID="0b554b5b-8ce8-4674-b7d3-235a7c0b777b" containerID="b70dda1f78d39cccfbb99a5cc673f5a0a00a41e15ec87c72827597290de2f613" exitCode=0
Sep 06 19:11:27.399708 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:27.399667    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4ncl" event={"ID":"0b554b5b-8ce8-4674-b7d3-235a7c0b777b","Type":"ContainerDied","Data":"b70dda1f78d39cccfbb99a5cc673f5a0a00a41e15ec87c72827597290de2f613"}
Sep 06 19:11:27.401498 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:27.401478    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs" event={"ID":"4557de3f-1c4f-448f-aec4-4552db157e3e","Type":"ContainerStarted","Data":"d148fb8b09216ebef285ce374a780967fb4ba7a6df299e4e771d5a07267e0d0d"}
Sep 06 19:11:27.404689 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:27.404649    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jm5gt" podStartSLOduration=3.8202974210000002 podStartE2EDuration="21.404631905s" totalImagesPullingTime="17.584334484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:11:27.404134174 +0000 UTC m=+21.585315979" watchObservedRunningTime="2026-09-06 19:11:27.404631905 +0000 UTC m=+21.585813711"
Sep 06 19:11:27.405670 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:27.405646    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tb692" event={"ID":"a21c0317-3ef0-4b2d-ae52-809097383928","Type":"ContainerStarted","Data":"5e6a4a83b138f71e01607181294bb30ef45718daf59577ec3d1e57e5e299baab"}
Sep 06 19:11:27.406987 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:27.406965    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8hj5v" event={"ID":"5c08a764-6c0b-48c6-a532-9d0b0b00ccb1","Type":"ContainerStarted","Data":"1e577724019dec70d4bd08b4cedb15d1a9f0508b57e8a67e74daa127134844c2"}
Sep 06 19:11:27.436015 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:27.435981    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-x75tt" podStartSLOduration=7.795176628 podStartE2EDuration="21.435969356s" totalImagesPullingTime="13.640792728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:11:27.435416607 +0000 UTC m=+21.616598418" watchObservedRunningTime="2026-09-06 19:11:27.435969356 +0000 UTC m=+21.617151161"
Sep 06 19:11:27.450310 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:27.450271    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.ec2.internal" podStartSLOduration=20.450261275 podStartE2EDuration="20.450261275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:11:27.450103311 +0000 UTC m=+21.631285118" watchObservedRunningTime="2026-09-06 19:11:27.450261275 +0000 UTC m=+21.631443081"
Sep 06 19:11:27.482996 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:27.481177    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8hj5v" podStartSLOduration=3.897735709 podStartE2EDuration="21.481158191s" totalImagesPullingTime="17.583422482s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:11:27.480229734 +0000 UTC m=+21.661411551" watchObservedRunningTime="2026-09-06 19:11:27.481158191 +0000 UTC m=+21.662339992"
Sep 06 19:11:27.482996 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:27.481530    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tb692" podStartSLOduration=3.727073474 podStartE2EDuration="21.481511219s" totalImagesPullingTime="17.754437745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:11:27.465487388 +0000 UTC m=+21.646669208" watchObservedRunningTime="2026-09-06 19:11:27.481511219 +0000 UTC m=+21.662693025"
Sep 06 19:11:27.909698 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:27.909492    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 06 19:11:28.297755 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:28.297656    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-06T19:11:27.909695161Z","UUID":"30d76e3c-664a-4456-8094-864a537b803f","Handler":null,"Name":"","Endpoint":""}
Sep 06 19:11:28.299472 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:28.299436    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 06 19:11:28.299472 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:28.299476    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 06 19:11:28.304048 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:28.304025    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:28.304168 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:28.304025    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:28.304168 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:28.304150    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlvsw" podUID="83a7379b-ab58-4439-998e-448e48150f3b"
Sep 06 19:11:28.304334 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:28.304025    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:28.304334 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:28.304197    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9h5h" podUID="52bd7358-7476-41f4-95da-5bdfefff01db"
Sep 06 19:11:28.304334 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:28.304239    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7crp9" podUID="dcd6e3bb-3d92-4fe9-8432-62ef628b2472"
Sep 06 19:11:28.411901 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:28.411868    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs" event={"ID":"4557de3f-1c4f-448f-aec4-4552db157e3e","Type":"ContainerStarted","Data":"046e4747f7ab780a46827eac9d3329b97fa20c6d29a03b4da11d8a0052c3a83d"}
Sep 06 19:11:28.413573 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:28.413522    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jdfqm" event={"ID":"317e1dba-5d84-4cfa-84aa-7d1a3dc399d5","Type":"ContainerStarted","Data":"89a2740db320cc89e814328bd8442266cd718523319b443d1918c53e3c53951b"}
Sep 06 19:11:28.428467 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:28.428414    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jdfqm" podStartSLOduration=4.865464697 podStartE2EDuration="22.428397892s" totalImagesPullingTime="17.562933195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:11:28.427847399 +0000 UTC m=+22.609029202" watchObservedRunningTime="2026-09-06 19:11:28.428397892 +0000 UTC m=+22.609579697"
Sep 06 19:11:29.271968 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:29.271928    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b5vzh_30fe2339-bf82-4be5-8cf4-f68f98014b4f/konnectivity-agent/0.log" limit=1024
Sep 06 19:11:29.353325 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:29.353294    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b5vzh_30fe2339-bf82-4be5-8cf4-f68f98014b4f/konnectivity-agent/0.log" limit=1024
Sep 06 19:11:29.418452 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:29.418366    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs" event={"ID":"4557de3f-1c4f-448f-aec4-4552db157e3e","Type":"ContainerStarted","Data":"1ced86d65afdff8557799055f55f6c2f9414bd124e3b135e5674aff7230af48c"}
Sep 06 19:11:29.421925 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:29.421905    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnssq_9254e6f1-b73d-49cc-8bc1-513925ccde4f/ovn-acl-logging/0.log"
Sep 06 19:11:29.422353 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:29.422316    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnssq" event={"ID":"9254e6f1-b73d-49cc-8bc1-513925ccde4f","Type":"ContainerStarted","Data":"50b06cb620a8688aa57391be34e5f92aa25e3c0ce89518b2f009dc2a57203b7e"}
Sep 06 19:11:29.433833 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:29.433789    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4ggs" podStartSLOduration=3.735683668 podStartE2EDuration="23.433774196s" totalImagesPullingTime="19.698090528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:11:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:11:29.433669632 +0000 UTC m=+23.614851439" watchObservedRunningTime="2026-09-06 19:11:29.433774196 +0000 UTC m=+23.614956002"
Sep 06 19:11:30.304951 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:30.304917    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:30.305133 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:30.304920    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:30.305133 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:30.305048    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:30.305133 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:30.305037    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9h5h" podUID="52bd7358-7476-41f4-95da-5bdfefff01db"
Sep 06 19:11:30.305265 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:30.305185    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlvsw" podUID="83a7379b-ab58-4439-998e-448e48150f3b"
Sep 06 19:11:30.305317 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:30.305262    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7crp9" podUID="dcd6e3bb-3d92-4fe9-8432-62ef628b2472"
Sep 06 19:11:30.544139 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:30.544108    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b5vzh"
Sep 06 19:11:30.544562 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:30.544233    2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 06 19:11:30.544740 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:30.544717    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b5vzh"
Sep 06 19:11:30.606800 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:30.606723    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b5vzh_30fe2339-bf82-4be5-8cf4-f68f98014b4f/konnectivity-agent/0.log" limit=1024
Sep 06 19:11:32.304514 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:32.304276    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:32.305006 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:32.304280    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:32.305006 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:32.304630    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7crp9" podUID="dcd6e3bb-3d92-4fe9-8432-62ef628b2472"
Sep 06 19:11:32.305006 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:32.304286    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:32.305006 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:32.304713    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlvsw" podUID="83a7379b-ab58-4439-998e-448e48150f3b"
Sep 06 19:11:32.305006 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:32.304753    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9h5h" podUID="52bd7358-7476-41f4-95da-5bdfefff01db"
Sep 06 19:11:32.431993 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:32.431968    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnssq_9254e6f1-b73d-49cc-8bc1-513925ccde4f/ovn-acl-logging/0.log"
Sep 06 19:11:32.432353 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:32.432334    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnssq" event={"ID":"9254e6f1-b73d-49cc-8bc1-513925ccde4f","Type":"ContainerStarted","Data":"eb50131940c516f794d3cbb31f5b3981614ebce3f3f71a58b14434571b20d5e0"}
Sep 06 19:11:32.432561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:32.432543    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:32.432781 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:32.432757    2566 scope.go:117] "RemoveContainer" containerID="87d6432950ae1c16486ab31f6b3da50daf2543dff2a2187b2081dfd49748b795"
Sep 06 19:11:32.434305 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:32.434285    2566 generic.go:356] "Generic (PLEG): container finished" podID="0b554b5b-8ce8-4674-b7d3-235a7c0b777b" containerID="cbc8f71691f08b053b036e20d18f8dd8321d5794c25acee40ffa9960aa170e5b" exitCode=0
Sep 06 19:11:32.434384 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:32.434324    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4ncl" event={"ID":"0b554b5b-8ce8-4674-b7d3-235a7c0b777b","Type":"ContainerDied","Data":"cbc8f71691f08b053b036e20d18f8dd8321d5794c25acee40ffa9960aa170e5b"}
Sep 06 19:11:32.448960 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:32.448941    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:33.377837 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:33.377811    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7crp9"]
Sep 06 19:11:33.378442 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:33.377982    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:33.378442 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:33.378082    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7crp9" podUID="dcd6e3bb-3d92-4fe9-8432-62ef628b2472"
Sep 06 19:11:33.381363 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:33.381337    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s9h5h"]
Sep 06 19:11:33.381726 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:33.381446    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:33.381726 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:33.381542    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9h5h" podUID="52bd7358-7476-41f4-95da-5bdfefff01db"
Sep 06 19:11:33.382014 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:33.381992    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qlvsw"]
Sep 06 19:11:33.382113 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:33.382100    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:33.382207 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:33.382186    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlvsw" podUID="83a7379b-ab58-4439-998e-448e48150f3b"
Sep 06 19:11:33.418384 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:33.418336    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:33.440012 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:33.439992    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnssq_9254e6f1-b73d-49cc-8bc1-513925ccde4f/ovn-acl-logging/0.log"
Sep 06 19:11:33.440383 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:33.440356    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnssq" event={"ID":"9254e6f1-b73d-49cc-8bc1-513925ccde4f","Type":"ContainerStarted","Data":"7724e9942574991c240ed00277f10576f3d6e458f3d377262c17322eba2b2089"}
Sep 06 19:11:33.440588 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:33.440569    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:33.442693 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:33.442668    2566 generic.go:356] "Generic (PLEG): container finished" podID="0b554b5b-8ce8-4674-b7d3-235a7c0b777b" containerID="60a0965837fa271938ca9728ef49e8e1b947f6cf6d2e8a80f303fd57891567a2" exitCode=0
Sep 06 19:11:33.442760 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:33.442711    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4ncl" event={"ID":"0b554b5b-8ce8-4674-b7d3-235a7c0b777b","Type":"ContainerDied","Data":"60a0965837fa271938ca9728ef49e8e1b947f6cf6d2e8a80f303fd57891567a2"}
Sep 06 19:11:33.455870 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:33.455850    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:11:33.468352 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:33.468315    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gnssq" podStartSLOduration=9.844847125 podStartE2EDuration="27.468302527s" totalImagesPullingTime="17.623455402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:11:33.466424171 +0000 UTC m=+27.647605977" watchObservedRunningTime="2026-09-06 19:11:33.468302527 +0000 UTC m=+27.649484330"
Sep 06 19:11:34.447303 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:34.447072    2566 generic.go:356] "Generic (PLEG): container finished" podID="0b554b5b-8ce8-4674-b7d3-235a7c0b777b" containerID="1c39aefd42035d5b6d79d95a6c082703841deae94cb47d85401841fe09ad74aa" exitCode=0
Sep 06 19:11:34.447687 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:34.447148    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4ncl" event={"ID":"0b554b5b-8ce8-4674-b7d3-235a7c0b777b","Type":"ContainerDied","Data":"1c39aefd42035d5b6d79d95a6c082703841deae94cb47d85401841fe09ad74aa"}
Sep 06 19:11:34.490434 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:34.490413    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b5vzh_30fe2339-bf82-4be5-8cf4-f68f98014b4f/konnectivity-agent/0.log" limit=1024
Sep 06 19:11:35.304278 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:35.304246    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:35.304278 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:35.304270    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:35.304504 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:35.304352    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:35.304504 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:35.304353    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9h5h" podUID="52bd7358-7476-41f4-95da-5bdfefff01db"
Sep 06 19:11:35.304504 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:35.304469    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7crp9" podUID="dcd6e3bb-3d92-4fe9-8432-62ef628b2472"
Sep 06 19:11:35.304658 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:35.304564    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlvsw" podUID="83a7379b-ab58-4439-998e-448e48150f3b"
Sep 06 19:11:37.304264 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:37.304231    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:37.304680 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:37.304329    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:37.304680 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:37.304337    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:37.304680 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:37.304454    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlvsw" podUID="83a7379b-ab58-4439-998e-448e48150f3b"
Sep 06 19:11:37.304680 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:37.304333    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9h5h" podUID="52bd7358-7476-41f4-95da-5bdfefff01db"
Sep 06 19:11:37.304680 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:37.304527    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7crp9" podUID="dcd6e3bb-3d92-4fe9-8432-62ef628b2472"
Sep 06 19:11:39.139839 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.139767    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-252.ec2.internal" event="NodeReady"
Sep 06 19:11:39.140240 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.139900    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 06 19:11:39.178462 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.178431    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gqxrr"]
Sep 06 19:11:39.181247 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.181231    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:11:39.181377 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.181321    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n9qkv"]
Sep 06 19:11:39.183789 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.183771    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 06 19:11:39.183887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.183771    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 06 19:11:39.183942 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.183771    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nxfsd\""
Sep 06 19:11:39.184003 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.183990    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n9qkv"
Sep 06 19:11:39.186398 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.186381    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 06 19:11:39.186718 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.186700    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 06 19:11:39.186800 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.186737    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 06 19:11:39.186955 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.186941    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qk5sj\""
Sep 06 19:11:39.190086 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.190069    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gqxrr"]
Sep 06 19:11:39.193053 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.193032    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n9qkv"]
Sep 06 19:11:39.281730 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.281707    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72bsp\" (UniqueName: \"kubernetes.io/projected/143de454-6f38-4836-a21c-5cac7ad78381-kube-api-access-72bsp\") pod \"ingress-canary-n9qkv\" (UID: \"143de454-6f38-4836-a21c-5cac7ad78381\") " pod="openshift-ingress-canary/ingress-canary-n9qkv"
Sep 06 19:11:39.281884 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.281742    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls\") pod \"dns-default-gqxrr\" (UID: \"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88\") " pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:11:39.281884 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.281800    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/143de454-6f38-4836-a21c-5cac7ad78381-cert\") pod \"ingress-canary-n9qkv\" (UID: \"143de454-6f38-4836-a21c-5cac7ad78381\") " pod="openshift-ingress-canary/ingress-canary-n9qkv"
Sep 06 19:11:39.281884 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.281842    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-config-volume\") pod \"dns-default-gqxrr\" (UID: \"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88\") " pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:11:39.281884 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.281876    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-tmp-dir\") pod \"dns-default-gqxrr\" (UID: \"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88\") " pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:11:39.282009 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.281900    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8mqc\" (UniqueName: \"kubernetes.io/projected/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-kube-api-access-f8mqc\") pod \"dns-default-gqxrr\" (UID: \"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88\") " pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:11:39.303901 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.303879    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:39.304039 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.303881    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:39.304094 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.303881    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:39.307367 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.307343    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 06 19:11:39.307768 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.307750    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 06 19:11:39.308003 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.307986    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 06 19:11:39.308227 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.308211    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-57svr\""
Sep 06 19:11:39.308438 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.308422    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6p7v7\""
Sep 06 19:11:39.308649 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.308634    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 06 19:11:39.382350 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.382327    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-config-volume\") pod \"dns-default-gqxrr\" (UID: \"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88\") " pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:11:39.382449 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.382357    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-tmp-dir\") pod \"dns-default-gqxrr\" (UID: \"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88\") " pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:11:39.382449 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.382385    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8mqc\" (UniqueName: \"kubernetes.io/projected/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-kube-api-access-f8mqc\") pod \"dns-default-gqxrr\" (UID: \"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88\") " pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:11:39.382449 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.382432    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72bsp\" (UniqueName: \"kubernetes.io/projected/143de454-6f38-4836-a21c-5cac7ad78381-kube-api-access-72bsp\") pod \"ingress-canary-n9qkv\" (UID: \"143de454-6f38-4836-a21c-5cac7ad78381\") " pod="openshift-ingress-canary/ingress-canary-n9qkv"
Sep 06 19:11:39.382631 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.382462    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls\") pod \"dns-default-gqxrr\" (UID: \"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88\") " pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:11:39.382631 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.382490    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/143de454-6f38-4836-a21c-5cac7ad78381-cert\") pod \"ingress-canary-n9qkv\" (UID: \"143de454-6f38-4836-a21c-5cac7ad78381\") " pod="openshift-ingress-canary/ingress-canary-n9qkv"
Sep 06 19:11:39.382631 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:39.382620    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 19:11:39.382778 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.382665    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-tmp-dir\") pod \"dns-default-gqxrr\" (UID: \"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88\") " pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:11:39.382778 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:39.382696    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls podName:a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88 nodeName:}" failed. No retries permitted until 2026-09-06 19:11:39.882676565 +0000 UTC m=+34.063858351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls") pod "dns-default-gqxrr" (UID: "a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88") : secret "dns-default-metrics-tls" not found
Sep 06 19:11:39.382942 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.382922    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-config-volume\") pod \"dns-default-gqxrr\" (UID: \"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88\") " pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:11:39.386700 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.386681    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/143de454-6f38-4836-a21c-5cac7ad78381-cert\") pod \"ingress-canary-n9qkv\" (UID: \"143de454-6f38-4836-a21c-5cac7ad78381\") " pod="openshift-ingress-canary/ingress-canary-n9qkv"
Sep 06 19:11:39.391039 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.390985    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8mqc\" (UniqueName: \"kubernetes.io/projected/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-kube-api-access-f8mqc\") pod \"dns-default-gqxrr\" (UID: \"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88\") " pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:11:39.391039 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.391029    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72bsp\" (UniqueName: \"kubernetes.io/projected/143de454-6f38-4836-a21c-5cac7ad78381-kube-api-access-72bsp\") pod \"ingress-canary-n9qkv\" (UID: \"143de454-6f38-4836-a21c-5cac7ad78381\") " pod="openshift-ingress-canary/ingress-canary-n9qkv"
Sep 06 19:11:39.462236 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.462204    2566 generic.go:356] "Generic (PLEG): container finished" podID="0b554b5b-8ce8-4674-b7d3-235a7c0b777b" containerID="acf7932310fc099660be998c8930fbb212aff9f866da387a1d33f38923e7130b" exitCode=0
Sep 06 19:11:39.462311 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.462277    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4ncl" event={"ID":"0b554b5b-8ce8-4674-b7d3-235a7c0b777b","Type":"ContainerDied","Data":"acf7932310fc099660be998c8930fbb212aff9f866da387a1d33f38923e7130b"}
Sep 06 19:11:39.498064 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.498044    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n9qkv"
Sep 06 19:11:39.778696 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.778664    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n9qkv"]
Sep 06 19:11:39.782187 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:39.782162    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143de454_6f38_4836_a21c_5cac7ad78381.slice/crio-a535de33f881c2687079428082e244b8ac978e247fc6c2f4732be25026cf3e83 WatchSource:0}: Error finding container a535de33f881c2687079428082e244b8ac978e247fc6c2f4732be25026cf3e83: Status 404 returned error can't find the container with id a535de33f881c2687079428082e244b8ac978e247fc6c2f4732be25026cf3e83
Sep 06 19:11:39.885577 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.885548    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls\") pod \"dns-default-gqxrr\" (UID: \"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88\") " pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:11:39.885719 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:39.885679    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 19:11:39.885757 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:39.885728    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls podName:a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88 nodeName:}" failed. No retries permitted until 2026-09-06 19:11:40.885714439 +0000 UTC m=+35.066896223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls") pod "dns-default-gqxrr" (UID: "a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88") : secret "dns-default-metrics-tls" not found
Sep 06 19:11:39.986706 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.986678    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-original-pull-secret\") pod \"global-pull-secret-syncer-7crp9\" (UID: \"dcd6e3bb-3d92-4fe9-8432-62ef628b2472\") " pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:39.989695 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:39.989675    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcd6e3bb-3d92-4fe9-8432-62ef628b2472-original-pull-secret\") pod \"global-pull-secret-syncer-7crp9\" (UID: \"dcd6e3bb-3d92-4fe9-8432-62ef628b2472\") " pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:40.087996 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:40.087921    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs\") pod \"network-metrics-daemon-qlvsw\" (UID: \"83a7379b-ab58-4439-998e-448e48150f3b\") " pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:11:40.088115 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:40.088081    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 06 19:11:40.088156 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:40.088149    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs podName:83a7379b-ab58-4439-998e-448e48150f3b nodeName:}" failed. No retries permitted until 2026-09-06 19:12:12.088130744 +0000 UTC m=+66.269312529 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs") pod "network-metrics-daemon-qlvsw" (UID: "83a7379b-ab58-4439-998e-448e48150f3b") : secret "metrics-daemon-secret" not found
Sep 06 19:11:40.188520 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:40.188480    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kvr2\" (UniqueName: \"kubernetes.io/projected/52bd7358-7476-41f4-95da-5bdfefff01db-kube-api-access-5kvr2\") pod \"network-check-target-s9h5h\" (UID: \"52bd7358-7476-41f4-95da-5bdfefff01db\") " pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:40.191206 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:40.191183    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kvr2\" (UniqueName: \"kubernetes.io/projected/52bd7358-7476-41f4-95da-5bdfefff01db-kube-api-access-5kvr2\") pod \"network-check-target-s9h5h\" (UID: \"52bd7358-7476-41f4-95da-5bdfefff01db\") " pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:40.215927 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:40.215904    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:40.226329 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:40.226307    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7crp9"
Sep 06 19:11:40.465722 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:40.465687    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n9qkv" event={"ID":"143de454-6f38-4836-a21c-5cac7ad78381","Type":"ContainerStarted","Data":"a535de33f881c2687079428082e244b8ac978e247fc6c2f4732be25026cf3e83"}
Sep 06 19:11:40.468509 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:40.468483    2566 generic.go:356] "Generic (PLEG): container finished" podID="0b554b5b-8ce8-4674-b7d3-235a7c0b777b" containerID="cdddf7bc00604a7be050968d0c302c4d15fac34b5cb6661f80a925971197dd00" exitCode=0
Sep 06 19:11:40.468631 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:40.468519    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4ncl" event={"ID":"0b554b5b-8ce8-4674-b7d3-235a7c0b777b","Type":"ContainerDied","Data":"cdddf7bc00604a7be050968d0c302c4d15fac34b5cb6661f80a925971197dd00"}
Sep 06 19:11:40.488412 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:40.488390    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s9h5h"]
Sep 06 19:11:40.493177 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:40.493153    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bd7358_7476_41f4_95da_5bdfefff01db.slice/crio-51b67750f9e091e7eca901ec5ef596e570a0777ec1e76c055a10c07e606466c8 WatchSource:0}: Error finding container 51b67750f9e091e7eca901ec5ef596e570a0777ec1e76c055a10c07e606466c8: Status 404 returned error can't find the container with id 51b67750f9e091e7eca901ec5ef596e570a0777ec1e76c055a10c07e606466c8
Sep 06 19:11:40.500266 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:40.500243    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7crp9"]
Sep 06 19:11:40.504046 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:11:40.504020    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd6e3bb_3d92_4fe9_8432_62ef628b2472.slice/crio-d6ff2413cb400d1268e1f59395784695743f37a1ae085d7147224eb8ca9ee82e WatchSource:0}: Error finding container d6ff2413cb400d1268e1f59395784695743f37a1ae085d7147224eb8ca9ee82e: Status 404 returned error can't find the container with id d6ff2413cb400d1268e1f59395784695743f37a1ae085d7147224eb8ca9ee82e
Sep 06 19:11:40.894194 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:40.894121    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls\") pod \"dns-default-gqxrr\" (UID: \"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88\") " pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:11:40.894320 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:40.894232    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 19:11:40.894320 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:40.894288    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls podName:a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88 nodeName:}" failed. No retries permitted until 2026-09-06 19:11:42.894273008 +0000 UTC m=+37.075454793 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls") pod "dns-default-gqxrr" (UID: "a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88") : secret "dns-default-metrics-tls" not found
Sep 06 19:11:41.473211 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:41.473173    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7crp9" event={"ID":"dcd6e3bb-3d92-4fe9-8432-62ef628b2472","Type":"ContainerStarted","Data":"d6ff2413cb400d1268e1f59395784695743f37a1ae085d7147224eb8ca9ee82e"}
Sep 06 19:11:41.474675 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:41.474643    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s9h5h" event={"ID":"52bd7358-7476-41f4-95da-5bdfefff01db","Type":"ContainerStarted","Data":"51b67750f9e091e7eca901ec5ef596e570a0777ec1e76c055a10c07e606466c8"}
Sep 06 19:11:41.479091 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:41.479065    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4ncl" event={"ID":"0b554b5b-8ce8-4674-b7d3-235a7c0b777b","Type":"ContainerStarted","Data":"8c9099268385c3b73958c8ca477c4876f7d91c7c9969ea8e1eeaf0ea5cbfc0a1"}
Sep 06 19:11:41.511991 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:41.511412    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q4ncl" podStartSLOduration=7.9083385360000005 podStartE2EDuration="35.511393126s" totalImagesPullingTime="27.60305459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:11:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:11:41.499086001 +0000 UTC m=+35.680267807" watchObservedRunningTime="2026-09-06 19:11:41.511393126 +0000 UTC m=+35.692574934"
Sep 06 19:11:42.910270 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:42.910038    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls\") pod \"dns-default-gqxrr\" (UID: \"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88\") " pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:11:42.910270 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:42.910185    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 19:11:42.910270 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:42.910246    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls podName:a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88 nodeName:}" failed. No retries permitted until 2026-09-06 19:11:46.910232933 +0000 UTC m=+41.091414716 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls") pod "dns-default-gqxrr" (UID: "a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88") : secret "dns-default-metrics-tls" not found
Sep 06 19:11:46.937115 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:46.937024    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls\") pod \"dns-default-gqxrr\" (UID: \"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88\") " pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:11:46.937469 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:46.937172    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 19:11:46.937469 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:46.937236    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls podName:a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88 nodeName:}" failed. No retries permitted until 2026-09-06 19:11:54.937219654 +0000 UTC m=+49.118401438 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls") pod "dns-default-gqxrr" (UID: "a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88") : secret "dns-default-metrics-tls" not found
Sep 06 19:11:47.496511 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:47.496474    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n9qkv" event={"ID":"143de454-6f38-4836-a21c-5cac7ad78381","Type":"ContainerStarted","Data":"238323b472fe7c2e4c4cb66b2429202f4bed5bc319daca089eb8af092f866cd2"}
Sep 06 19:11:47.498071 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:47.498038    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s9h5h" event={"ID":"52bd7358-7476-41f4-95da-5bdfefff01db","Type":"ContainerStarted","Data":"4a21142c69558aeeec506d05e65e376bd73cc137b2e802b87d5eb0289b618ed8"}
Sep 06 19:11:47.498196 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:47.498136    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:11:47.499373 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:47.499353    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7crp9" event={"ID":"dcd6e3bb-3d92-4fe9-8432-62ef628b2472","Type":"ContainerStarted","Data":"f4a067f266b5250fb634619c5f122574cd4061f6923f234b910319cd06ebcee7"}
Sep 06 19:11:47.512161 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:47.512123    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n9qkv" podStartSLOduration=1.8170723629999999 podStartE2EDuration="8.512112195s" totalImagesPullingTime="6.695039832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:11:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:11:47.510733378 +0000 UTC m=+41.691915185" watchObservedRunningTime="2026-09-06 19:11:47.512112195 +0000 UTC m=+41.693294040"
Sep 06 19:11:47.524345 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:47.524308    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-s9h5h" podStartSLOduration=35.540547502 podStartE2EDuration="41.524299556s" totalImagesPullingTime="5.983752054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:11:47.52353042 +0000 UTC m=+41.704712226" watchObservedRunningTime="2026-09-06 19:11:47.524299556 +0000 UTC m=+41.705481362"
Sep 06 19:11:47.536782 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:47.536745    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7crp9" podStartSLOduration=35.553754866 podStartE2EDuration="41.536734468s" totalImagesPullingTime="5.982979602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:11:47.536232865 +0000 UTC m=+41.717414672" watchObservedRunningTime="2026-09-06 19:11:47.536734468 +0000 UTC m=+41.717916275"
Sep 06 19:11:54.984661 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:11:54.984598    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls\") pod \"dns-default-gqxrr\" (UID: \"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88\") " pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:11:54.985127 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:54.984724    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 19:11:54.985127 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:11:54.984812    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls podName:a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88 nodeName:}" failed. No retries permitted until 2026-09-06 19:12:10.984786828 +0000 UTC m=+65.165968632 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls") pod "dns-default-gqxrr" (UID: "a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88") : secret "dns-default-metrics-tls" not found
Sep 06 19:12:05.461136 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:12:05.461105    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnssq"
Sep 06 19:12:11.084640 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:12:11.084588    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls\") pod \"dns-default-gqxrr\" (UID: \"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88\") " pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:12:11.085090 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:12:11.084717    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 19:12:11.085090 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:12:11.084773    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls podName:a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88 nodeName:}" failed. No retries permitted until 2026-09-06 19:12:43.084757103 +0000 UTC m=+97.265938905 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls") pod "dns-default-gqxrr" (UID: "a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88") : secret "dns-default-metrics-tls" not found
Sep 06 19:12:12.092128 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:12:12.092087    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs\") pod \"network-metrics-daemon-qlvsw\" (UID: \"83a7379b-ab58-4439-998e-448e48150f3b\") " pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:12:12.092494 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:12:12.092198    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 06 19:12:12.092494 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:12:12.092262    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs podName:83a7379b-ab58-4439-998e-448e48150f3b nodeName:}" failed. No retries permitted until 2026-09-06 19:13:16.092247617 +0000 UTC m=+130.273429401 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs") pod "network-metrics-daemon-qlvsw" (UID: "83a7379b-ab58-4439-998e-448e48150f3b") : secret "metrics-daemon-secret" not found
Sep 06 19:12:18.503987 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:12:18.503949    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-s9h5h"
Sep 06 19:12:43.097017 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:12:43.096967    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls\") pod \"dns-default-gqxrr\" (UID: \"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88\") " pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:12:43.097395 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:12:43.097115    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 19:12:43.097395 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:12:43.097188    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls podName:a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88 nodeName:}" failed. No retries permitted until 2026-09-06 19:13:47.097169859 +0000 UTC m=+161.278351647 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls") pod "dns-default-gqxrr" (UID: "a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88") : secret "dns-default-metrics-tls" not found
Sep 06 19:13:16.098802 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:13:16.098759    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs\") pod \"network-metrics-daemon-qlvsw\" (UID: \"83a7379b-ab58-4439-998e-448e48150f3b\") " pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:13:16.099296 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:13:16.098869    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 06 19:13:16.099296 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:13:16.098940    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs podName:83a7379b-ab58-4439-998e-448e48150f3b nodeName:}" failed. No retries permitted until 2026-09-06 19:15:18.09892049 +0000 UTC m=+252.280102287 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs") pod "network-metrics-daemon-qlvsw" (UID: "83a7379b-ab58-4439-998e-448e48150f3b") : secret "metrics-daemon-secret" not found
Sep 06 19:13:42.191200 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:13:42.191149    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-gqxrr" podUID="a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88"
Sep 06 19:13:42.320992 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:13:42.320957    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-qlvsw" podUID="83a7379b-ab58-4439-998e-448e48150f3b"
Sep 06 19:13:42.773502 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:13:42.773470    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:13:47.196517 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:13:47.196473    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls\") pod \"dns-default-gqxrr\" (UID: \"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88\") " pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:13:47.197364 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:13:47.196643    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 19:13:47.197364 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:13:47.196703    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls podName:a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88 nodeName:}" failed. No retries permitted until 2026-09-06 19:15:49.196687822 +0000 UTC m=+283.377869606 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls") pod "dns-default-gqxrr" (UID: "a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88") : secret "dns-default-metrics-tls" not found
Sep 06 19:13:54.304072 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:13:54.304036    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:14:59.960029 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.959984    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-djlz4"]
Sep 06 19:14:59.962921 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.962898    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-59sch"]
Sep 06 19:14:59.963066 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.963051    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4"
Sep 06 19:14:59.965433 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.965415    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d45bfdd8d-96984"]
Sep 06 19:14:59.965581 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.965565    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-59sch"
Sep 06 19:14:59.965791 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.965771    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 06 19:14:59.966353 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.966334    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 06 19:14:59.966431 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.966354    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-7wpm9\""
Sep 06 19:14:59.966659 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.966645    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 06 19:14:59.967451 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.967436    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 06 19:14:59.968223 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.968208    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:14:59.969147 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.969129    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 06 19:14:59.970290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.970271    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 06 19:14:59.970505 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.970489    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 06 19:14:59.970580 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.970568    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-877x7\""
Sep 06 19:14:59.970645 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.970594    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 06 19:14:59.971737 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.971720    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 06 19:14:59.972109 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.972091    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 06 19:14:59.972109 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.972103    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ptdbl\""
Sep 06 19:14:59.972274 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.972244    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 06 19:14:59.973123 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.973104    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 06 19:14:59.974461 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.974440    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-59sch"]
Sep 06 19:14:59.977087 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.976822    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 06 19:14:59.978676 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.978656    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-djlz4"]
Sep 06 19:14:59.981187 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:14:59.981171    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d45bfdd8d-96984"]
Sep 06 19:15:00.054422 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.054384    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lh7gj"]
Sep 06 19:15:00.057234 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.057209    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-72v7v"]
Sep 06 19:15:00.057354 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.057338    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lh7gj"
Sep 06 19:15:00.059563 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.059544    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 06 19:15:00.059872 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.059859    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-72v7v"
Sep 06 19:15:00.060141 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.060122    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 06 19:15:00.060420 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.060405    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zn5zp\""
Sep 06 19:15:00.060570 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.060555    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/033d0561-8f3c-4e3c-afad-88da2108830e-serving-cert\") pod \"insights-operator-76bcd6ccf5-djlz4\" (UID: \"033d0561-8f3c-4e3c-afad-88da2108830e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4"
Sep 06 19:15:00.060630 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.060580    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/033d0561-8f3c-4e3c-afad-88da2108830e-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-djlz4\" (UID: \"033d0561-8f3c-4e3c-afad-88da2108830e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4"
Sep 06 19:15:00.060630 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.060599    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1f1f8330-fe91-47ea-9ea4-766dff7f934e-ca-trust-extracted\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.060630 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.060630    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/033d0561-8f3c-4e3c-afad-88da2108830e-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-djlz4\" (UID: \"033d0561-8f3c-4e3c-afad-88da2108830e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4"
Sep 06 19:15:00.060737 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.060686    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1f1f8330-fe91-47ea-9ea4-766dff7f934e-installation-pull-secrets\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.060737 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.060707    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffa3473d-8821-4d6b-b633-c9b7a28f7285-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-59sch\" (UID: \"ffa3473d-8821-4d6b-b633-c9b7a28f7285\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-59sch"
Sep 06 19:15:00.060737 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.060731    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f1f8330-fe91-47ea-9ea4-766dff7f934e-trusted-ca\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.060826 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.060757    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1f1f8330-fe91-47ea-9ea4-766dff7f934e-image-registry-private-configuration\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.060826 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.060774    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqz8v\" (UniqueName: \"kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-kube-api-access-sqz8v\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.060826 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.060800    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mflct\" (UniqueName: \"kubernetes.io/projected/033d0561-8f3c-4e3c-afad-88da2108830e-kube-api-access-mflct\") pod \"insights-operator-76bcd6ccf5-djlz4\" (UID: \"033d0561-8f3c-4e3c-afad-88da2108830e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4"
Sep 06 19:15:00.060912 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.060832    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/033d0561-8f3c-4e3c-afad-88da2108830e-tmp\") pod \"insights-operator-76bcd6ccf5-djlz4\" (UID: \"033d0561-8f3c-4e3c-afad-88da2108830e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4"
Sep 06 19:15:00.060912 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.060855    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ffa3473d-8821-4d6b-b633-c9b7a28f7285-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-59sch\" (UID: \"ffa3473d-8821-4d6b-b633-c9b7a28f7285\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-59sch"
Sep 06 19:15:00.060912 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.060874    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb8h2\" (UniqueName: \"kubernetes.io/projected/ffa3473d-8821-4d6b-b633-c9b7a28f7285-kube-api-access-wb8h2\") pod \"cluster-monitoring-operator-6cdcc96965-59sch\" (UID: \"ffa3473d-8821-4d6b-b633-c9b7a28f7285\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-59sch"
Sep 06 19:15:00.060912 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.060897    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-tls\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.061026 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.060911    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-certificates\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.061026 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.060925    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-bound-sa-token\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.061026 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.060939    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/033d0561-8f3c-4e3c-afad-88da2108830e-snapshots\") pod \"insights-operator-76bcd6ccf5-djlz4\" (UID: \"033d0561-8f3c-4e3c-afad-88da2108830e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4"
Sep 06 19:15:00.062005 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.061989    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 06 19:15:00.062042 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.062007    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 06 19:15:00.062232 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.062216    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 06 19:15:00.062264 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.062241    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9fss6\""
Sep 06 19:15:00.068758 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.068734    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lh7gj"]
Sep 06 19:15:00.070036 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.070016    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-72v7v"]
Sep 06 19:15:00.161618 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.161578    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/033d0561-8f3c-4e3c-afad-88da2108830e-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-djlz4\" (UID: \"033d0561-8f3c-4e3c-afad-88da2108830e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4"
Sep 06 19:15:00.161801 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.161635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1f1f8330-fe91-47ea-9ea4-766dff7f934e-ca-trust-extracted\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.161801 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.161660    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvqxl\" (UniqueName: \"kubernetes.io/projected/54d49606-bc52-481e-a3f7-bf0f3b60ab2c-kube-api-access-jvqxl\") pod \"volume-data-source-validator-695c895bd6-lh7gj\" (UID: \"54d49606-bc52-481e-a3f7-bf0f3b60ab2c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lh7gj"
Sep 06 19:15:00.161801 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.161688    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/033d0561-8f3c-4e3c-afad-88da2108830e-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-djlz4\" (UID: \"033d0561-8f3c-4e3c-afad-88da2108830e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4"
Sep 06 19:15:00.161801 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.161716    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1f1f8330-fe91-47ea-9ea4-766dff7f934e-installation-pull-secrets\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.161801 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.161743    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffa3473d-8821-4d6b-b633-c9b7a28f7285-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-59sch\" (UID: \"ffa3473d-8821-4d6b-b633-c9b7a28f7285\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-59sch"
Sep 06 19:15:00.161801 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.161768    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f1f8330-fe91-47ea-9ea4-766dff7f934e-trusted-ca\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.162114 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.161808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1f1f8330-fe91-47ea-9ea4-766dff7f934e-image-registry-private-configuration\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.162114 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.161834    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqz8v\" (UniqueName: \"kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-kube-api-access-sqz8v\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.162114 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.161856    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mflct\" (UniqueName: \"kubernetes.io/projected/033d0561-8f3c-4e3c-afad-88da2108830e-kube-api-access-mflct\") pod \"insights-operator-76bcd6ccf5-djlz4\" (UID: \"033d0561-8f3c-4e3c-afad-88da2108830e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4"
Sep 06 19:15:00.162114 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:00.161870    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 06 19:15:00.162114 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.161885    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/033d0561-8f3c-4e3c-afad-88da2108830e-tmp\") pod \"insights-operator-76bcd6ccf5-djlz4\" (UID: \"033d0561-8f3c-4e3c-afad-88da2108830e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4"
Sep 06 19:15:00.162114 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.161908    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2baeb20-1961-4452-8da2-f314f9d39364-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-72v7v\" (UID: \"e2baeb20-1961-4452-8da2-f314f9d39364\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-72v7v"
Sep 06 19:15:00.162114 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.161934    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jqhp\" (UniqueName: \"kubernetes.io/projected/e2baeb20-1961-4452-8da2-f314f9d39364-kube-api-access-6jqhp\") pod \"cluster-samples-operator-7cdcfcf76c-72v7v\" (UID: \"e2baeb20-1961-4452-8da2-f314f9d39364\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-72v7v"
Sep 06 19:15:00.162114 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:00.161957    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa3473d-8821-4d6b-b633-c9b7a28f7285-cluster-monitoring-operator-tls podName:ffa3473d-8821-4d6b-b633-c9b7a28f7285 nodeName:}" failed. No retries permitted until 2026-09-06 19:15:00.661937049 +0000 UTC m=+234.843118847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ffa3473d-8821-4d6b-b633-c9b7a28f7285-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-59sch" (UID: "ffa3473d-8821-4d6b-b633-c9b7a28f7285") : secret "cluster-monitoring-operator-tls" not found
Sep 06 19:15:00.162114 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.162052    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1f1f8330-fe91-47ea-9ea4-766dff7f934e-ca-trust-extracted\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.162569 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.162118    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ffa3473d-8821-4d6b-b633-c9b7a28f7285-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-59sch\" (UID: \"ffa3473d-8821-4d6b-b633-c9b7a28f7285\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-59sch"
Sep 06 19:15:00.162569 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.162146    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb8h2\" (UniqueName: \"kubernetes.io/projected/ffa3473d-8821-4d6b-b633-c9b7a28f7285-kube-api-access-wb8h2\") pod \"cluster-monitoring-operator-6cdcc96965-59sch\" (UID: \"ffa3473d-8821-4d6b-b633-c9b7a28f7285\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-59sch"
Sep 06 19:15:00.162569 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.162178    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-tls\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.162569 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.162200    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-certificates\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.162569 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.162223    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-bound-sa-token\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.162569 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.162246    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/033d0561-8f3c-4e3c-afad-88da2108830e-snapshots\") pod \"insights-operator-76bcd6ccf5-djlz4\" (UID: \"033d0561-8f3c-4e3c-afad-88da2108830e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4"
Sep 06 19:15:00.162569 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.162274    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/033d0561-8f3c-4e3c-afad-88da2108830e-serving-cert\") pod \"insights-operator-76bcd6ccf5-djlz4\" (UID: \"033d0561-8f3c-4e3c-afad-88da2108830e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4"
Sep 06 19:15:00.162569 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.162292    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/033d0561-8f3c-4e3c-afad-88da2108830e-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-djlz4\" (UID: \"033d0561-8f3c-4e3c-afad-88da2108830e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4"
Sep 06 19:15:00.162569 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:00.162569    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 19:15:00.163057 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:00.162585    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d45bfdd8d-96984: secret "image-registry-tls" not found
Sep 06 19:15:00.163057 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.162663    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/033d0561-8f3c-4e3c-afad-88da2108830e-tmp\") pod \"insights-operator-76bcd6ccf5-djlz4\" (UID: \"033d0561-8f3c-4e3c-afad-88da2108830e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4"
Sep 06 19:15:00.163057 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:00.162681    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-tls podName:1f1f8330-fe91-47ea-9ea4-766dff7f934e nodeName:}" failed. No retries permitted until 2026-09-06 19:15:00.662663429 +0000 UTC m=+234.843845213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-tls") pod "image-registry-d45bfdd8d-96984" (UID: "1f1f8330-fe91-47ea-9ea4-766dff7f934e") : secret "image-registry-tls" not found
Sep 06 19:15:00.163057 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.162722    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/033d0561-8f3c-4e3c-afad-88da2108830e-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-djlz4\" (UID: \"033d0561-8f3c-4e3c-afad-88da2108830e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4"
Sep 06 19:15:00.163057 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.162902    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f1f8330-fe91-47ea-9ea4-766dff7f934e-trusted-ca\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.163363 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.163340    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-certificates\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.163415 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.163367    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ffa3473d-8821-4d6b-b633-c9b7a28f7285-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-59sch\" (UID: \"ffa3473d-8821-4d6b-b633-c9b7a28f7285\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-59sch"
Sep 06 19:15:00.163415 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.163348    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/033d0561-8f3c-4e3c-afad-88da2108830e-snapshots\") pod \"insights-operator-76bcd6ccf5-djlz4\" (UID: \"033d0561-8f3c-4e3c-afad-88da2108830e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4"
Sep 06 19:15:00.165572 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.165546    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1f1f8330-fe91-47ea-9ea4-766dff7f934e-image-registry-private-configuration\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.165905 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.165885    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/033d0561-8f3c-4e3c-afad-88da2108830e-serving-cert\") pod \"insights-operator-76bcd6ccf5-djlz4\" (UID: \"033d0561-8f3c-4e3c-afad-88da2108830e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4"
Sep 06 19:15:00.165986 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.165946    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1f1f8330-fe91-47ea-9ea4-766dff7f934e-installation-pull-secrets\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.170422 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.170402    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mflct\" (UniqueName: \"kubernetes.io/projected/033d0561-8f3c-4e3c-afad-88da2108830e-kube-api-access-mflct\") pod \"insights-operator-76bcd6ccf5-djlz4\" (UID: \"033d0561-8f3c-4e3c-afad-88da2108830e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4"
Sep 06 19:15:00.170872 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.170853    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqz8v\" (UniqueName: \"kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-kube-api-access-sqz8v\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.171058 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.171037    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb8h2\" (UniqueName: \"kubernetes.io/projected/ffa3473d-8821-4d6b-b633-c9b7a28f7285-kube-api-access-wb8h2\") pod \"cluster-monitoring-operator-6cdcc96965-59sch\" (UID: \"ffa3473d-8821-4d6b-b633-c9b7a28f7285\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-59sch"
Sep 06 19:15:00.171241 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.171225    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-bound-sa-token\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.263532 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.263439    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvqxl\" (UniqueName: \"kubernetes.io/projected/54d49606-bc52-481e-a3f7-bf0f3b60ab2c-kube-api-access-jvqxl\") pod \"volume-data-source-validator-695c895bd6-lh7gj\" (UID: \"54d49606-bc52-481e-a3f7-bf0f3b60ab2c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lh7gj"
Sep 06 19:15:00.263532 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.263509    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2baeb20-1961-4452-8da2-f314f9d39364-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-72v7v\" (UID: \"e2baeb20-1961-4452-8da2-f314f9d39364\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-72v7v"
Sep 06 19:15:00.263532 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.263525    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jqhp\" (UniqueName: \"kubernetes.io/projected/e2baeb20-1961-4452-8da2-f314f9d39364-kube-api-access-6jqhp\") pod \"cluster-samples-operator-7cdcfcf76c-72v7v\" (UID: \"e2baeb20-1961-4452-8da2-f314f9d39364\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-72v7v"
Sep 06 19:15:00.263776 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:00.263627    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 06 19:15:00.263776 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:00.263692    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2baeb20-1961-4452-8da2-f314f9d39364-samples-operator-tls podName:e2baeb20-1961-4452-8da2-f314f9d39364 nodeName:}" failed. No retries permitted until 2026-09-06 19:15:00.763676509 +0000 UTC m=+234.944858293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e2baeb20-1961-4452-8da2-f314f9d39364-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-72v7v" (UID: "e2baeb20-1961-4452-8da2-f314f9d39364") : secret "samples-operator-tls" not found
Sep 06 19:15:00.271477 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.271445    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvqxl\" (UniqueName: \"kubernetes.io/projected/54d49606-bc52-481e-a3f7-bf0f3b60ab2c-kube-api-access-jvqxl\") pod \"volume-data-source-validator-695c895bd6-lh7gj\" (UID: \"54d49606-bc52-481e-a3f7-bf0f3b60ab2c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lh7gj"
Sep 06 19:15:00.271599 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.271527    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jqhp\" (UniqueName: \"kubernetes.io/projected/e2baeb20-1961-4452-8da2-f314f9d39364-kube-api-access-6jqhp\") pod \"cluster-samples-operator-7cdcfcf76c-72v7v\" (UID: \"e2baeb20-1961-4452-8da2-f314f9d39364\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-72v7v"
Sep 06 19:15:00.273282 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.273267    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4"
Sep 06 19:15:00.367225 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.367190    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lh7gj"
Sep 06 19:15:00.541566 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.541490    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-djlz4"]
Sep 06 19:15:00.544489 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:15:00.544466    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033d0561_8f3c_4e3c_afad_88da2108830e.slice/crio-aa8296b756926382fedc2effdfd1b1427a5aabff9917a40b59fe3fc05650f599 WatchSource:0}: Error finding container aa8296b756926382fedc2effdfd1b1427a5aabff9917a40b59fe3fc05650f599: Status 404 returned error can't find the container with id aa8296b756926382fedc2effdfd1b1427a5aabff9917a40b59fe3fc05650f599
Sep 06 19:15:00.631594 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.631565    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lh7gj"]
Sep 06 19:15:00.634566 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:15:00.634546    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d49606_bc52_481e_a3f7_bf0f3b60ab2c.slice/crio-7cf078e43f917a42d8cafe0f0cdacc620b703768b3554de8ea290ca0ae151e1a WatchSource:0}: Error finding container 7cf078e43f917a42d8cafe0f0cdacc620b703768b3554de8ea290ca0ae151e1a: Status 404 returned error can't find the container with id 7cf078e43f917a42d8cafe0f0cdacc620b703768b3554de8ea290ca0ae151e1a
Sep 06 19:15:00.666013 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.665986    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-tls\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:00.666114 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.666046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffa3473d-8821-4d6b-b633-c9b7a28f7285-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-59sch\" (UID: \"ffa3473d-8821-4d6b-b633-c9b7a28f7285\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-59sch"
Sep 06 19:15:00.666177 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:00.666136    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 19:15:00.666177 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:00.666148    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d45bfdd8d-96984: secret "image-registry-tls" not found
Sep 06 19:15:00.666271 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:00.666177    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 06 19:15:00.666271 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:00.666200    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-tls podName:1f1f8330-fe91-47ea-9ea4-766dff7f934e nodeName:}" failed. No retries permitted until 2026-09-06 19:15:01.666182176 +0000 UTC m=+235.847363980 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-tls") pod "image-registry-d45bfdd8d-96984" (UID: "1f1f8330-fe91-47ea-9ea4-766dff7f934e") : secret "image-registry-tls" not found
Sep 06 19:15:00.666271 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:00.666234    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa3473d-8821-4d6b-b633-c9b7a28f7285-cluster-monitoring-operator-tls podName:ffa3473d-8821-4d6b-b633-c9b7a28f7285 nodeName:}" failed. No retries permitted until 2026-09-06 19:15:01.666217615 +0000 UTC m=+235.847399415 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ffa3473d-8821-4d6b-b633-c9b7a28f7285-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-59sch" (UID: "ffa3473d-8821-4d6b-b633-c9b7a28f7285") : secret "cluster-monitoring-operator-tls" not found
Sep 06 19:15:00.766662 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.766628    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2baeb20-1961-4452-8da2-f314f9d39364-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-72v7v\" (UID: \"e2baeb20-1961-4452-8da2-f314f9d39364\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-72v7v"
Sep 06 19:15:00.766822 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:00.766801    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 06 19:15:00.766891 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:00.766884    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2baeb20-1961-4452-8da2-f314f9d39364-samples-operator-tls podName:e2baeb20-1961-4452-8da2-f314f9d39364 nodeName:}" failed. No retries permitted until 2026-09-06 19:15:01.766861782 +0000 UTC m=+235.948043587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e2baeb20-1961-4452-8da2-f314f9d39364-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-72v7v" (UID: "e2baeb20-1961-4452-8da2-f314f9d39364") : secret "samples-operator-tls" not found
Sep 06 19:15:00.962213 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.962178    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lh7gj" event={"ID":"54d49606-bc52-481e-a3f7-bf0f3b60ab2c","Type":"ContainerStarted","Data":"7cf078e43f917a42d8cafe0f0cdacc620b703768b3554de8ea290ca0ae151e1a"}
Sep 06 19:15:00.963259 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:00.963237    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4" event={"ID":"033d0561-8f3c-4e3c-afad-88da2108830e","Type":"ContainerStarted","Data":"aa8296b756926382fedc2effdfd1b1427a5aabff9917a40b59fe3fc05650f599"}
Sep 06 19:15:01.674275 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:01.674229    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffa3473d-8821-4d6b-b633-c9b7a28f7285-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-59sch\" (UID: \"ffa3473d-8821-4d6b-b633-c9b7a28f7285\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-59sch"
Sep 06 19:15:01.674451 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:01.674404    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 06 19:15:01.674529 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:01.674449    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-tls\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:01.674529 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:01.674476    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa3473d-8821-4d6b-b633-c9b7a28f7285-cluster-monitoring-operator-tls podName:ffa3473d-8821-4d6b-b633-c9b7a28f7285 nodeName:}" failed. No retries permitted until 2026-09-06 19:15:03.674460264 +0000 UTC m=+237.855642049 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ffa3473d-8821-4d6b-b633-c9b7a28f7285-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-59sch" (UID: "ffa3473d-8821-4d6b-b633-c9b7a28f7285") : secret "cluster-monitoring-operator-tls" not found
Sep 06 19:15:01.674670 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:01.674548    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 19:15:01.674670 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:01.674561    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d45bfdd8d-96984: secret "image-registry-tls" not found
Sep 06 19:15:01.674670 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:01.674629    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-tls podName:1f1f8330-fe91-47ea-9ea4-766dff7f934e nodeName:}" failed. No retries permitted until 2026-09-06 19:15:03.674596099 +0000 UTC m=+237.855777898 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-tls") pod "image-registry-d45bfdd8d-96984" (UID: "1f1f8330-fe91-47ea-9ea4-766dff7f934e") : secret "image-registry-tls" not found
Sep 06 19:15:01.775476 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:01.775443    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2baeb20-1961-4452-8da2-f314f9d39364-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-72v7v\" (UID: \"e2baeb20-1961-4452-8da2-f314f9d39364\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-72v7v"
Sep 06 19:15:01.775689 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:01.775630    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 06 19:15:01.775758 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:01.775692    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2baeb20-1961-4452-8da2-f314f9d39364-samples-operator-tls podName:e2baeb20-1961-4452-8da2-f314f9d39364 nodeName:}" failed. No retries permitted until 2026-09-06 19:15:03.775677034 +0000 UTC m=+237.956858830 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e2baeb20-1961-4452-8da2-f314f9d39364-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-72v7v" (UID: "e2baeb20-1961-4452-8da2-f314f9d39364") : secret "samples-operator-tls" not found
Sep 06 19:15:02.974205 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:02.974167    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lh7gj" event={"ID":"54d49606-bc52-481e-a3f7-bf0f3b60ab2c","Type":"ContainerStarted","Data":"cc383b970a3d328bf51a5ced64bb7c96cb53d8907a4aee6c98f4aee804c84275"}
Sep 06 19:15:02.975841 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:02.975807    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4" event={"ID":"033d0561-8f3c-4e3c-afad-88da2108830e","Type":"ContainerStarted","Data":"dfa69f95e3f1e28e5157e426a3643c9b1c018b150e61f0b83a074ff092ab0f37"}
Sep 06 19:15:02.990172 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:02.990124    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lh7gj" podStartSLOduration=1.013492706 podStartE2EDuration="2.990111396s" totalImagesPullingTime="1.97661869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:15:02.989078563 +0000 UTC m=+237.170260370" watchObservedRunningTime="2026-09-06 19:15:02.990111396 +0000 UTC m=+237.171293201"
Sep 06 19:15:03.004469 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:03.004431    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4" podStartSLOduration=1.93271354 podStartE2EDuration="4.004416838s" totalImagesPullingTime="2.071703298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:15:03.004271359 +0000 UTC m=+237.185453167" watchObservedRunningTime="2026-09-06 19:15:03.004416838 +0000 UTC m=+237.185598642"
Sep 06 19:15:03.692120 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:03.692079    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffa3473d-8821-4d6b-b633-c9b7a28f7285-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-59sch\" (UID: \"ffa3473d-8821-4d6b-b633-c9b7a28f7285\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-59sch"
Sep 06 19:15:03.692302 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:03.692194    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-tls\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:03.692302 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:03.692256    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 06 19:15:03.692413 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:03.692332    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 19:15:03.692413 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:03.692349    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d45bfdd8d-96984: secret "image-registry-tls" not found
Sep 06 19:15:03.692413 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:03.692334    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa3473d-8821-4d6b-b633-c9b7a28f7285-cluster-monitoring-operator-tls podName:ffa3473d-8821-4d6b-b633-c9b7a28f7285 nodeName:}" failed. No retries permitted until 2026-09-06 19:15:07.692314009 +0000 UTC m=+241.873495809 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ffa3473d-8821-4d6b-b633-c9b7a28f7285-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-59sch" (UID: "ffa3473d-8821-4d6b-b633-c9b7a28f7285") : secret "cluster-monitoring-operator-tls" not found
Sep 06 19:15:03.692413 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:03.692409    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-tls podName:1f1f8330-fe91-47ea-9ea4-766dff7f934e nodeName:}" failed. No retries permitted until 2026-09-06 19:15:07.692393493 +0000 UTC m=+241.873575472 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-tls") pod "image-registry-d45bfdd8d-96984" (UID: "1f1f8330-fe91-47ea-9ea4-766dff7f934e") : secret "image-registry-tls" not found
Sep 06 19:15:03.751596 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:03.751561    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kjj7s"]
Sep 06 19:15:03.754697 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:03.754676    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:03.757100 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:03.757079    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 06 19:15:03.757193 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:03.757148    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 06 19:15:03.758086 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:03.758070    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-grwwn\""
Sep 06 19:15:03.764774 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:03.764756    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kjj7s"]
Sep 06 19:15:03.793063 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:03.793044    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2baeb20-1961-4452-8da2-f314f9d39364-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-72v7v\" (UID: \"e2baeb20-1961-4452-8da2-f314f9d39364\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-72v7v"
Sep 06 19:15:03.793150 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:03.793132    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 06 19:15:03.793185 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:03.793177    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2baeb20-1961-4452-8da2-f314f9d39364-samples-operator-tls podName:e2baeb20-1961-4452-8da2-f314f9d39364 nodeName:}" failed. No retries permitted until 2026-09-06 19:15:07.79316455 +0000 UTC m=+241.974346334 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e2baeb20-1961-4452-8da2-f314f9d39364-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-72v7v" (UID: "e2baeb20-1961-4452-8da2-f314f9d39364") : secret "samples-operator-tls" not found
Sep 06 19:15:03.894036 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:03.894003    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:03.894036 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:03.894036    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:03.894259 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:03.894091    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfpb6\" (UniqueName: \"kubernetes.io/projected/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-kube-api-access-sfpb6\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:03.894259 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:03.894122    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-data-volume\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:03.894259 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:03.894141    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-crio-socket\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:03.994926 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:03.994894    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:03.994926 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:03.994926    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:03.995333 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:03.994969    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfpb6\" (UniqueName: \"kubernetes.io/projected/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-kube-api-access-sfpb6\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:03.995333 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:03.995005    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-data-volume\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:03.995333 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:03.995023    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-crio-socket\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:03.995333 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:03.995022    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 19:15:03.995333 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:03.995096    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-crio-socket\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:03.995333 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:03.995101    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-kube-rbac-proxy-cm podName:ede8a3c0-14ed-4a64-beda-caa3b4c5d601 nodeName:}" failed. No retries permitted until 2026-09-06 19:15:04.495085077 +0000 UTC m=+238.676266861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kjj7s" (UID: "ede8a3c0-14ed-4a64-beda-caa3b4c5d601") : configmap "kube-rbac-proxy" not found
Sep 06 19:15:03.995333 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:03.995027    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 19:15:03.995333 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:03.995183    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-insights-runtime-extractor-tls podName:ede8a3c0-14ed-4a64-beda-caa3b4c5d601 nodeName:}" failed. No retries permitted until 2026-09-06 19:15:04.495167824 +0000 UTC m=+238.676349620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kjj7s" (UID: "ede8a3c0-14ed-4a64-beda-caa3b4c5d601") : secret "insights-runtime-extractor-tls" not found
Sep 06 19:15:03.995333 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:03.995329    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-data-volume\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:04.004098 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:04.004070    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfpb6\" (UniqueName: \"kubernetes.io/projected/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-kube-api-access-sfpb6\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:04.499178 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:04.499144    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:04.499329 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:04.499182    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:04.499329 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:04.499260    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 19:15:04.499329 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:04.499288    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 19:15:04.499329 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:04.499319    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-kube-rbac-proxy-cm podName:ede8a3c0-14ed-4a64-beda-caa3b4c5d601 nodeName:}" failed. No retries permitted until 2026-09-06 19:15:05.499305486 +0000 UTC m=+239.680487270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kjj7s" (UID: "ede8a3c0-14ed-4a64-beda-caa3b4c5d601") : configmap "kube-rbac-proxy" not found
Sep 06 19:15:04.499464 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:04.499338    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-insights-runtime-extractor-tls podName:ede8a3c0-14ed-4a64-beda-caa3b4c5d601 nodeName:}" failed. No retries permitted until 2026-09-06 19:15:05.499325882 +0000 UTC m=+239.680507666 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kjj7s" (UID: "ede8a3c0-14ed-4a64-beda-caa3b4c5d601") : secret "insights-runtime-extractor-tls" not found
Sep 06 19:15:05.507424 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:05.507385    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:05.507424 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:05.507422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:05.507914 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:05.507531    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 19:15:05.507914 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:05.507539    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 19:15:05.507914 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:05.507590    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-kube-rbac-proxy-cm podName:ede8a3c0-14ed-4a64-beda-caa3b4c5d601 nodeName:}" failed. No retries permitted until 2026-09-06 19:15:07.507572948 +0000 UTC m=+241.688754748 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kjj7s" (UID: "ede8a3c0-14ed-4a64-beda-caa3b4c5d601") : configmap "kube-rbac-proxy" not found
Sep 06 19:15:05.507914 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:05.507623    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-insights-runtime-extractor-tls podName:ede8a3c0-14ed-4a64-beda-caa3b4c5d601 nodeName:}" failed. No retries permitted until 2026-09-06 19:15:07.507598267 +0000 UTC m=+241.688780052 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kjj7s" (UID: "ede8a3c0-14ed-4a64-beda-caa3b4c5d601") : secret "insights-runtime-extractor-tls" not found
Sep 06 19:15:05.715658 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:05.715627    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jm5gt_176abf0b-9b68-46d3-a657-ab219939488a/dns-node-resolver/0.log"
Sep 06 19:15:06.915643 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:06.915615    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x75tt_a68be042-a345-44a5-9564-1d6d16d11b8a/node-ca/0.log"
Sep 06 19:15:07.524002 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:07.523962    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:07.524002 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:07.524003    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:07.524200 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:07.524113    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 19:15:07.524200 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:07.524142    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 19:15:07.524200 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:07.524181    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-kube-rbac-proxy-cm podName:ede8a3c0-14ed-4a64-beda-caa3b4c5d601 nodeName:}" failed. No retries permitted until 2026-09-06 19:15:11.5241632 +0000 UTC m=+245.705344984 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kjj7s" (UID: "ede8a3c0-14ed-4a64-beda-caa3b4c5d601") : configmap "kube-rbac-proxy" not found
Sep 06 19:15:07.524200 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:07.524197    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-insights-runtime-extractor-tls podName:ede8a3c0-14ed-4a64-beda-caa3b4c5d601 nodeName:}" failed. No retries permitted until 2026-09-06 19:15:11.524190575 +0000 UTC m=+245.705372359 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kjj7s" (UID: "ede8a3c0-14ed-4a64-beda-caa3b4c5d601") : secret "insights-runtime-extractor-tls" not found
Sep 06 19:15:07.717776 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:07.717743    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n9qkv_143de454-6f38-4836-a21c-5cac7ad78381/serve-healthcheck-canary/0.log"
Sep 06 19:15:07.724944 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:07.724921    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffa3473d-8821-4d6b-b633-c9b7a28f7285-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-59sch\" (UID: \"ffa3473d-8821-4d6b-b633-c9b7a28f7285\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-59sch"
Sep 06 19:15:07.725066 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:07.725008    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-tls\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:07.725108 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:07.725070    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 06 19:15:07.725108 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:07.725094    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 19:15:07.725181 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:07.725115    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d45bfdd8d-96984: secret "image-registry-tls" not found
Sep 06 19:15:07.725181 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:07.725128    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa3473d-8821-4d6b-b633-c9b7a28f7285-cluster-monitoring-operator-tls podName:ffa3473d-8821-4d6b-b633-c9b7a28f7285 nodeName:}" failed. No retries permitted until 2026-09-06 19:15:15.725112641 +0000 UTC m=+249.906294426 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ffa3473d-8821-4d6b-b633-c9b7a28f7285-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-59sch" (UID: "ffa3473d-8821-4d6b-b633-c9b7a28f7285") : secret "cluster-monitoring-operator-tls" not found
Sep 06 19:15:07.725181 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:07.725158    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-tls podName:1f1f8330-fe91-47ea-9ea4-766dff7f934e nodeName:}" failed. No retries permitted until 2026-09-06 19:15:15.725144971 +0000 UTC m=+249.906326756 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-tls") pod "image-registry-d45bfdd8d-96984" (UID: "1f1f8330-fe91-47ea-9ea4-766dff7f934e") : secret "image-registry-tls" not found
Sep 06 19:15:07.734329 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:07.734307    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-5fgbh"]
Sep 06 19:15:07.738518 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:07.738499    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-5fgbh"
Sep 06 19:15:07.740908 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:07.740885    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 06 19:15:07.741015 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:07.740999    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-blbbn\""
Sep 06 19:15:07.742041 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:07.742021    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 06 19:15:07.742041 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:07.742035    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 06 19:15:07.742190 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:07.742044    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 06 19:15:07.743569 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:07.743552    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-5fgbh"]
Sep 06 19:15:07.825691 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:07.825596    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2baeb20-1961-4452-8da2-f314f9d39364-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-72v7v\" (UID: \"e2baeb20-1961-4452-8da2-f314f9d39364\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-72v7v"
Sep 06 19:15:07.825820 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:07.825712    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4fd4a413-198b-4f42-88d0-8d31bd369e5a-signing-cabundle\") pod \"service-ca-75d99c484-5fgbh\" (UID: \"4fd4a413-198b-4f42-88d0-8d31bd369e5a\") " pod="openshift-service-ca/service-ca-75d99c484-5fgbh"
Sep 06 19:15:07.825820 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:07.825735    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4fd4a413-198b-4f42-88d0-8d31bd369e5a-signing-key\") pod \"service-ca-75d99c484-5fgbh\" (UID: \"4fd4a413-198b-4f42-88d0-8d31bd369e5a\") " pod="openshift-service-ca/service-ca-75d99c484-5fgbh"
Sep 06 19:15:07.825820 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:07.825749    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqc2p\" (UniqueName: \"kubernetes.io/projected/4fd4a413-198b-4f42-88d0-8d31bd369e5a-kube-api-access-zqc2p\") pod \"service-ca-75d99c484-5fgbh\" (UID: \"4fd4a413-198b-4f42-88d0-8d31bd369e5a\") " pod="openshift-service-ca/service-ca-75d99c484-5fgbh"
Sep 06 19:15:07.825820 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:07.825755    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 06 19:15:07.825820 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:07.825816    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2baeb20-1961-4452-8da2-f314f9d39364-samples-operator-tls podName:e2baeb20-1961-4452-8da2-f314f9d39364 nodeName:}" failed. No retries permitted until 2026-09-06 19:15:15.825796451 +0000 UTC m=+250.006978253 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e2baeb20-1961-4452-8da2-f314f9d39364-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-72v7v" (UID: "e2baeb20-1961-4452-8da2-f314f9d39364") : secret "samples-operator-tls" not found
Sep 06 19:15:07.926089 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:07.926060    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4fd4a413-198b-4f42-88d0-8d31bd369e5a-signing-cabundle\") pod \"service-ca-75d99c484-5fgbh\" (UID: \"4fd4a413-198b-4f42-88d0-8d31bd369e5a\") " pod="openshift-service-ca/service-ca-75d99c484-5fgbh"
Sep 06 19:15:07.926089 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:07.926090    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4fd4a413-198b-4f42-88d0-8d31bd369e5a-signing-key\") pod \"service-ca-75d99c484-5fgbh\" (UID: \"4fd4a413-198b-4f42-88d0-8d31bd369e5a\") " pod="openshift-service-ca/service-ca-75d99c484-5fgbh"
Sep 06 19:15:07.926416 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:07.926106    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqc2p\" (UniqueName: \"kubernetes.io/projected/4fd4a413-198b-4f42-88d0-8d31bd369e5a-kube-api-access-zqc2p\") pod \"service-ca-75d99c484-5fgbh\" (UID: \"4fd4a413-198b-4f42-88d0-8d31bd369e5a\") " pod="openshift-service-ca/service-ca-75d99c484-5fgbh"
Sep 06 19:15:07.926820 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:07.926803    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4fd4a413-198b-4f42-88d0-8d31bd369e5a-signing-cabundle\") pod \"service-ca-75d99c484-5fgbh\" (UID: \"4fd4a413-198b-4f42-88d0-8d31bd369e5a\") " pod="openshift-service-ca/service-ca-75d99c484-5fgbh"
Sep 06 19:15:07.928597 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:07.928579    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4fd4a413-198b-4f42-88d0-8d31bd369e5a-signing-key\") pod \"service-ca-75d99c484-5fgbh\" (UID: \"4fd4a413-198b-4f42-88d0-8d31bd369e5a\") " pod="openshift-service-ca/service-ca-75d99c484-5fgbh"
Sep 06 19:15:07.934543 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:07.934519    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqc2p\" (UniqueName: \"kubernetes.io/projected/4fd4a413-198b-4f42-88d0-8d31bd369e5a-kube-api-access-zqc2p\") pod \"service-ca-75d99c484-5fgbh\" (UID: \"4fd4a413-198b-4f42-88d0-8d31bd369e5a\") " pod="openshift-service-ca/service-ca-75d99c484-5fgbh"
Sep 06 19:15:08.047538 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:08.047516    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-5fgbh"
Sep 06 19:15:08.312532 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:08.312502    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-5fgbh"]
Sep 06 19:15:08.315336 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:15:08.315311    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd4a413_198b_4f42_88d0_8d31bd369e5a.slice/crio-f78311ec30e18b304a57856163a49e6033beb44367f0a9fdd6042cc319c6c771 WatchSource:0}: Error finding container f78311ec30e18b304a57856163a49e6033beb44367f0a9fdd6042cc319c6c771: Status 404 returned error can't find the container with id f78311ec30e18b304a57856163a49e6033beb44367f0a9fdd6042cc319c6c771
Sep 06 19:15:08.994213 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:08.994177    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-5fgbh" event={"ID":"4fd4a413-198b-4f42-88d0-8d31bd369e5a","Type":"ContainerStarted","Data":"f78311ec30e18b304a57856163a49e6033beb44367f0a9fdd6042cc319c6c771"}
Sep 06 19:15:11.002778 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:11.002730    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-5fgbh" event={"ID":"4fd4a413-198b-4f42-88d0-8d31bd369e5a","Type":"ContainerStarted","Data":"a969ce39b4ba4a04e1445d14c05cdd860cd27fe7b124cd33b0e1dc36cc86141a"}
Sep 06 19:15:11.017573 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:11.017526    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-5fgbh" podStartSLOduration=2.262087217 podStartE2EDuration="4.017510673s" totalImagesPullingTime="1.755423456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:15:11.016687176 +0000 UTC m=+245.197868983" watchObservedRunningTime="2026-09-06 19:15:11.017510673 +0000 UTC m=+245.198692478"
Sep 06 19:15:11.553886 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:11.553751    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:11.553886 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:11.553805    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:11.554111 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:11.553914    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 19:15:11.554111 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:11.553992    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-kube-rbac-proxy-cm podName:ede8a3c0-14ed-4a64-beda-caa3b4c5d601 nodeName:}" failed. No retries permitted until 2026-09-06 19:15:19.553971976 +0000 UTC m=+253.735153765 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kjj7s" (UID: "ede8a3c0-14ed-4a64-beda-caa3b4c5d601") : configmap "kube-rbac-proxy" not found
Sep 06 19:15:11.554381 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:11.554366    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 19:15:11.554445 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:11.554429    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-insights-runtime-extractor-tls podName:ede8a3c0-14ed-4a64-beda-caa3b4c5d601 nodeName:}" failed. No retries permitted until 2026-09-06 19:15:19.554414426 +0000 UTC m=+253.735596213 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kjj7s" (UID: "ede8a3c0-14ed-4a64-beda-caa3b4c5d601") : secret "insights-runtime-extractor-tls" not found
Sep 06 19:15:15.783934 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:15.783890    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-tls\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:15.784400 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:15.783987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffa3473d-8821-4d6b-b633-c9b7a28f7285-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-59sch\" (UID: \"ffa3473d-8821-4d6b-b633-c9b7a28f7285\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-59sch"
Sep 06 19:15:15.784400 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:15.784099    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 06 19:15:15.784400 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:15.784167    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa3473d-8821-4d6b-b633-c9b7a28f7285-cluster-monitoring-operator-tls podName:ffa3473d-8821-4d6b-b633-c9b7a28f7285 nodeName:}" failed. No retries permitted until 2026-09-06 19:15:31.784149124 +0000 UTC m=+265.965330911 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ffa3473d-8821-4d6b-b633-c9b7a28f7285-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-59sch" (UID: "ffa3473d-8821-4d6b-b633-c9b7a28f7285") : secret "cluster-monitoring-operator-tls" not found
Sep 06 19:15:15.786385 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:15.786359    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-tls\") pod \"image-registry-d45bfdd8d-96984\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") " pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:15.884643 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:15.884590    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2baeb20-1961-4452-8da2-f314f9d39364-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-72v7v\" (UID: \"e2baeb20-1961-4452-8da2-f314f9d39364\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-72v7v"
Sep 06 19:15:15.887076 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:15.887047    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2baeb20-1961-4452-8da2-f314f9d39364-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-72v7v\" (UID: \"e2baeb20-1961-4452-8da2-f314f9d39364\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-72v7v"
Sep 06 19:15:15.888333 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:15.888314    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ptdbl\""
Sep 06 19:15:15.896651 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:15.896633    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:15.976298 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:15.976266    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9fss6\""
Sep 06 19:15:15.984390 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:15.984369    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-72v7v"
Sep 06 19:15:16.164256 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:16.164189    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d45bfdd8d-96984"]
Sep 06 19:15:16.166712 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:15:16.166688    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1f8330_fe91_47ea_9ea4_766dff7f934e.slice/crio-2d065a787b58e2b2d29db58f82f03dc3fd0ce58b121da7a7d966ff3b736b4f04 WatchSource:0}: Error finding container 2d065a787b58e2b2d29db58f82f03dc3fd0ce58b121da7a7d966ff3b736b4f04: Status 404 returned error can't find the container with id 2d065a787b58e2b2d29db58f82f03dc3fd0ce58b121da7a7d966ff3b736b4f04
Sep 06 19:15:16.250629 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:16.250581    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-72v7v"]
Sep 06 19:15:17.021758 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:17.021719    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-72v7v" event={"ID":"e2baeb20-1961-4452-8da2-f314f9d39364","Type":"ContainerStarted","Data":"7cfc02fbdc2b90edb57919a7639587602482a11903f2e1639507d640bf3281ff"}
Sep 06 19:15:17.023421 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:17.023392    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d45bfdd8d-96984" event={"ID":"1f1f8330-fe91-47ea-9ea4-766dff7f934e","Type":"ContainerStarted","Data":"d923390e905122cc5bf66c220db53d5fe181712a72cc98bb704a0bd05ad11e7d"}
Sep 06 19:15:17.023550 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:17.023433    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d45bfdd8d-96984" event={"ID":"1f1f8330-fe91-47ea-9ea4-766dff7f934e","Type":"ContainerStarted","Data":"2d065a787b58e2b2d29db58f82f03dc3fd0ce58b121da7a7d966ff3b736b4f04"}
Sep 06 19:15:17.023550 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:17.023532    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:17.041955 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:17.041910    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d45bfdd8d-96984" podStartSLOduration=18.041890989 podStartE2EDuration="18.041890989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:15:17.040897903 +0000 UTC m=+251.222079710" watchObservedRunningTime="2026-09-06 19:15:17.041890989 +0000 UTC m=+251.223072795"
Sep 06 19:15:18.028112 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:18.028074    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-72v7v" event={"ID":"e2baeb20-1961-4452-8da2-f314f9d39364","Type":"ContainerStarted","Data":"f3cd12e058c2a2b92ba38c53f5502eb42a85dc003554e69f419493a86f3e7bc1"}
Sep 06 19:15:18.028459 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:18.028139    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-72v7v" event={"ID":"e2baeb20-1961-4452-8da2-f314f9d39364","Type":"ContainerStarted","Data":"aa9ce06afd261a515fb8c4ebb595183dba9d70f60bde7f37996f2b0f8a379f09"}
Sep 06 19:15:18.043159 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:18.043118    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-72v7v" podStartSLOduration=16.486021963 podStartE2EDuration="18.043103003s" totalImagesPullingTime="1.55708104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:15:18.041449737 +0000 UTC m=+252.222631542" watchObservedRunningTime="2026-09-06 19:15:18.043103003 +0000 UTC m=+252.224284810"
Sep 06 19:15:18.101996 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:18.101960    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs\") pod \"network-metrics-daemon-qlvsw\" (UID: \"83a7379b-ab58-4439-998e-448e48150f3b\") " pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:15:18.102151 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:18.102092    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 06 19:15:18.102151 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:18.102149    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs podName:83a7379b-ab58-4439-998e-448e48150f3b nodeName:}" failed. No retries permitted until 2026-09-06 19:17:20.102129998 +0000 UTC m=+374.283311799 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs") pod "network-metrics-daemon-qlvsw" (UID: "83a7379b-ab58-4439-998e-448e48150f3b") : secret "metrics-daemon-secret" not found
Sep 06 19:15:19.616307 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:19.616256    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:19.616307 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:19.616308    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:19.616845 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:19.616424    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 19:15:19.616845 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:19.616511    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-insights-runtime-extractor-tls podName:ede8a3c0-14ed-4a64-beda-caa3b4c5d601 nodeName:}" failed. No retries permitted until 2026-09-06 19:15:35.616489866 +0000 UTC m=+269.797671650 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kjj7s" (UID: "ede8a3c0-14ed-4a64-beda-caa3b4c5d601") : secret "insights-runtime-extractor-tls" not found
Sep 06 19:15:19.616942 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:19.616853    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:28.757170 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.757096    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d45bfdd8d-96984"]
Sep 06 19:15:28.786617 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.786583    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6dcf6d58c-nd6k8"]
Sep 06 19:15:28.789461 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.789444    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.800391 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.800370    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dcf6d58c-nd6k8"]
Sep 06 19:15:28.885667 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.885642    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22089faf-55a4-4003-849a-56580918f07b-image-registry-private-configuration\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.885793 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.885679    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22089faf-55a4-4003-849a-56580918f07b-bound-sa-token\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.885793 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.885698    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22089faf-55a4-4003-849a-56580918f07b-trusted-ca\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.885793 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.885717    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22089faf-55a4-4003-849a-56580918f07b-registry-tls\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.885894 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.885827    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22089faf-55a4-4003-849a-56580918f07b-installation-pull-secrets\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.885894 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.885866    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22089faf-55a4-4003-849a-56580918f07b-registry-certificates\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.885955 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.885898    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wfp4\" (UniqueName: \"kubernetes.io/projected/22089faf-55a4-4003-849a-56580918f07b-kube-api-access-6wfp4\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.885955 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.885934    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22089faf-55a4-4003-849a-56580918f07b-ca-trust-extracted\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.986610 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.986586    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22089faf-55a4-4003-849a-56580918f07b-registry-tls\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.986720 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.986670    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22089faf-55a4-4003-849a-56580918f07b-installation-pull-secrets\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.986784 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.986762    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22089faf-55a4-4003-849a-56580918f07b-registry-certificates\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.986820 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.986810    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wfp4\" (UniqueName: \"kubernetes.io/projected/22089faf-55a4-4003-849a-56580918f07b-kube-api-access-6wfp4\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.986863 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.986849    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22089faf-55a4-4003-849a-56580918f07b-ca-trust-extracted\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.986896 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.986882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22089faf-55a4-4003-849a-56580918f07b-image-registry-private-configuration\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.986945 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.986919    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22089faf-55a4-4003-849a-56580918f07b-bound-sa-token\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.987116 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.987095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22089faf-55a4-4003-849a-56580918f07b-trusted-ca\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.987240 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.987219    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22089faf-55a4-4003-849a-56580918f07b-ca-trust-extracted\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.987863 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.987835    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22089faf-55a4-4003-849a-56580918f07b-trusted-ca\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.987863 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.987850    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22089faf-55a4-4003-849a-56580918f07b-registry-certificates\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.989246 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.989229    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22089faf-55a4-4003-849a-56580918f07b-installation-pull-secrets\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.989355 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.989333    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22089faf-55a4-4003-849a-56580918f07b-registry-tls\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.989416 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.989401    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22089faf-55a4-4003-849a-56580918f07b-image-registry-private-configuration\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.996655 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.996632    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22089faf-55a4-4003-849a-56580918f07b-bound-sa-token\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:28.996730 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:28.996700    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wfp4\" (UniqueName: \"kubernetes.io/projected/22089faf-55a4-4003-849a-56580918f07b-kube-api-access-6wfp4\") pod \"image-registry-6dcf6d58c-nd6k8\" (UID: \"22089faf-55a4-4003-849a-56580918f07b\") " pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:29.097670 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:29.097595    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:29.365869 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:29.365793    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dcf6d58c-nd6k8"]
Sep 06 19:15:29.369763 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:15:29.369740    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22089faf_55a4_4003_849a_56580918f07b.slice/crio-01d3423690e13a88b6a2a42e7a89c14e093fb1a87ecc3edd3cdd8d375f48b768 WatchSource:0}: Error finding container 01d3423690e13a88b6a2a42e7a89c14e093fb1a87ecc3edd3cdd8d375f48b768: Status 404 returned error can't find the container with id 01d3423690e13a88b6a2a42e7a89c14e093fb1a87ecc3edd3cdd8d375f48b768
Sep 06 19:15:30.064678 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:30.064638    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8" event={"ID":"22089faf-55a4-4003-849a-56580918f07b","Type":"ContainerStarted","Data":"0f76b86b48f87f23fdc298f9eaf7339d50348aa03334275e4ba6278892daf39b"}
Sep 06 19:15:30.065130 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:30.064695    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8" event={"ID":"22089faf-55a4-4003-849a-56580918f07b","Type":"ContainerStarted","Data":"01d3423690e13a88b6a2a42e7a89c14e093fb1a87ecc3edd3cdd8d375f48b768"}
Sep 06 19:15:30.065130 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:30.064749    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:15:30.083562 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:30.083515    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8" podStartSLOduration=2.083500105 podStartE2EDuration="2.083500105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:15:30.082633792 +0000 UTC m=+264.263815595" watchObservedRunningTime="2026-09-06 19:15:30.083500105 +0000 UTC m=+264.264681911"
Sep 06 19:15:31.813147 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:31.813108    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffa3473d-8821-4d6b-b633-c9b7a28f7285-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-59sch\" (UID: \"ffa3473d-8821-4d6b-b633-c9b7a28f7285\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-59sch"
Sep 06 19:15:31.815562 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:31.815541    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffa3473d-8821-4d6b-b633-c9b7a28f7285-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-59sch\" (UID: \"ffa3473d-8821-4d6b-b633-c9b7a28f7285\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-59sch"
Sep 06 19:15:32.082797 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:32.082728    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-877x7\""
Sep 06 19:15:32.091660 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:32.091642    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-59sch"
Sep 06 19:15:32.404689 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:32.404596    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-59sch"]
Sep 06 19:15:32.408023 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:15:32.407999    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa3473d_8821_4d6b_b633_c9b7a28f7285.slice/crio-fd020061f5e3e0b347f6200c66601f1c4035501871ea5dcc3bb022ed7163b88c WatchSource:0}: Error finding container fd020061f5e3e0b347f6200c66601f1c4035501871ea5dcc3bb022ed7163b88c: Status 404 returned error can't find the container with id fd020061f5e3e0b347f6200c66601f1c4035501871ea5dcc3bb022ed7163b88c
Sep 06 19:15:33.075529 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:33.075488    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-59sch" event={"ID":"ffa3473d-8821-4d6b-b633-c9b7a28f7285","Type":"ContainerStarted","Data":"fd020061f5e3e0b347f6200c66601f1c4035501871ea5dcc3bb022ed7163b88c"}
Sep 06 19:15:35.084334 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:35.084295    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-59sch" event={"ID":"ffa3473d-8821-4d6b-b633-c9b7a28f7285","Type":"ContainerStarted","Data":"1ef393bd5c8065b241e6abb2dfd631e78439daa9bb9a64f91b6f32727f5e85c6"}
Sep 06 19:15:35.101742 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:35.101676    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-59sch" podStartSLOduration=34.27168708 podStartE2EDuration="36.10166137s" totalImagesPullingTime="1.82997429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:15:35.101358262 +0000 UTC m=+269.282540068" watchObservedRunningTime="2026-09-06 19:15:35.10166137 +0000 UTC m=+269.282843168"
Sep 06 19:15:35.646099 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:35.646059    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:35.648489 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:35.648469    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ede8a3c0-14ed-4a64-beda-caa3b4c5d601-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjj7s\" (UID: \"ede8a3c0-14ed-4a64-beda-caa3b4c5d601\") " pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:35.866270 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:35.866234    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-grwwn\""
Sep 06 19:15:35.874084 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:35.874064    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kjj7s"
Sep 06 19:15:36.143132 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:36.143104    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kjj7s"]
Sep 06 19:15:37.092894 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:37.092817    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kjj7s" event={"ID":"ede8a3c0-14ed-4a64-beda-caa3b4c5d601","Type":"ContainerStarted","Data":"4f691987342d842e718ce8a571fe067cbe9c050779f7836385ac2bd0d51cb1b8"}
Sep 06 19:15:37.093039 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:37.092898    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kjj7s" event={"ID":"ede8a3c0-14ed-4a64-beda-caa3b4c5d601","Type":"ContainerStarted","Data":"7c98dded1ebc5a8d7bae1477d0120dafc9a4dbfc826d50198be8f9a4d5fafd75"}
Sep 06 19:15:37.093039 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:37.092918    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kjj7s" event={"ID":"ede8a3c0-14ed-4a64-beda-caa3b4c5d601","Type":"ContainerStarted","Data":"68e66d543d21451d4eab807fbe978f76df04b76fc4f753524aa8bc9e3bb526ba"}
Sep 06 19:15:38.097456 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:38.097417    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kjj7s" event={"ID":"ede8a3c0-14ed-4a64-beda-caa3b4c5d601","Type":"ContainerStarted","Data":"7854143580c192c074f4dcce480e1bf34171ad07f7835fa65db708f32fd59c01"}
Sep 06 19:15:38.114064 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:38.114018    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kjj7s" podStartSLOduration=33.381296595 podStartE2EDuration="35.114002873s" totalImagesPullingTime="1.732706278s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:15:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:15:38.112687727 +0000 UTC m=+272.293869535" watchObservedRunningTime="2026-09-06 19:15:38.114002873 +0000 UTC m=+272.295184701"
Sep 06 19:15:38.754477 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:38.754443    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-jwt4v"]
Sep 06 19:15:38.757584 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:38.757562    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-jwt4v"
Sep 06 19:15:38.759929 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:38.759907    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 06 19:15:38.760054 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:38.759942    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 06 19:15:38.761107 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:38.761076    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-2n4g9\""
Sep 06 19:15:38.761210 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:38.761112    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 06 19:15:38.763518 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:38.763452    2566 patch_prober.go:28] interesting pod/image-registry-d45bfdd8d-96984 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 06 19:15:38.763518 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:38.763500    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-d45bfdd8d-96984" podUID="1f1f8330-fe91-47ea-9ea4-766dff7f934e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 06 19:15:38.768188 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:38.768169    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-jwt4v"]
Sep 06 19:15:38.872925 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:38.872893    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb73d373-f92d-4c73-bb27-61e4192eb39b-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-jwt4v\" (UID: \"fb73d373-f92d-4c73-bb27-61e4192eb39b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jwt4v"
Sep 06 19:15:38.872925 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:38.872925    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb73d373-f92d-4c73-bb27-61e4192eb39b-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-jwt4v\" (UID: \"fb73d373-f92d-4c73-bb27-61e4192eb39b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jwt4v"
Sep 06 19:15:38.873121 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:38.872977    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fb73d373-f92d-4c73-bb27-61e4192eb39b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-jwt4v\" (UID: \"fb73d373-f92d-4c73-bb27-61e4192eb39b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jwt4v"
Sep 06 19:15:38.873121 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:38.873048    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k962v\" (UniqueName: \"kubernetes.io/projected/fb73d373-f92d-4c73-bb27-61e4192eb39b-kube-api-access-k962v\") pod \"prometheus-operator-6f4946db56-jwt4v\" (UID: \"fb73d373-f92d-4c73-bb27-61e4192eb39b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jwt4v"
Sep 06 19:15:38.974270 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:38.974239    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fb73d373-f92d-4c73-bb27-61e4192eb39b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-jwt4v\" (UID: \"fb73d373-f92d-4c73-bb27-61e4192eb39b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jwt4v"
Sep 06 19:15:38.974404 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:38.974277    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k962v\" (UniqueName: \"kubernetes.io/projected/fb73d373-f92d-4c73-bb27-61e4192eb39b-kube-api-access-k962v\") pod \"prometheus-operator-6f4946db56-jwt4v\" (UID: \"fb73d373-f92d-4c73-bb27-61e4192eb39b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jwt4v"
Sep 06 19:15:38.974404 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:38.974336    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb73d373-f92d-4c73-bb27-61e4192eb39b-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-jwt4v\" (UID: \"fb73d373-f92d-4c73-bb27-61e4192eb39b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jwt4v"
Sep 06 19:15:38.974404 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:38.974363    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb73d373-f92d-4c73-bb27-61e4192eb39b-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-jwt4v\" (UID: \"fb73d373-f92d-4c73-bb27-61e4192eb39b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jwt4v"
Sep 06 19:15:38.975477 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:38.975457    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb73d373-f92d-4c73-bb27-61e4192eb39b-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-jwt4v\" (UID: \"fb73d373-f92d-4c73-bb27-61e4192eb39b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jwt4v"
Sep 06 19:15:38.976786 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:38.976763    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fb73d373-f92d-4c73-bb27-61e4192eb39b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-jwt4v\" (UID: \"fb73d373-f92d-4c73-bb27-61e4192eb39b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jwt4v"
Sep 06 19:15:38.976868 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:38.976772    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb73d373-f92d-4c73-bb27-61e4192eb39b-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-jwt4v\" (UID: \"fb73d373-f92d-4c73-bb27-61e4192eb39b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jwt4v"
Sep 06 19:15:38.982534 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:38.982518    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k962v\" (UniqueName: \"kubernetes.io/projected/fb73d373-f92d-4c73-bb27-61e4192eb39b-kube-api-access-k962v\") pod \"prometheus-operator-6f4946db56-jwt4v\" (UID: \"fb73d373-f92d-4c73-bb27-61e4192eb39b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jwt4v"
Sep 06 19:15:39.068533 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:39.068474    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-jwt4v"
Sep 06 19:15:39.337095 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:39.337021    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-jwt4v"]
Sep 06 19:15:39.339577 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:15:39.339550    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb73d373_f92d_4c73_bb27_61e4192eb39b.slice/crio-054392f8c94f201039e179f0fc2c41247c0c70c8ca758901e36f9e529321aa41 WatchSource:0}: Error finding container 054392f8c94f201039e179f0fc2c41247c0c70c8ca758901e36f9e529321aa41: Status 404 returned error can't find the container with id 054392f8c94f201039e179f0fc2c41247c0c70c8ca758901e36f9e529321aa41
Sep 06 19:15:40.104771 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:40.104733    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-jwt4v" event={"ID":"fb73d373-f92d-4c73-bb27-61e4192eb39b","Type":"ContainerStarted","Data":"054392f8c94f201039e179f0fc2c41247c0c70c8ca758901e36f9e529321aa41"}
Sep 06 19:15:41.109370 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:41.109338    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-jwt4v" event={"ID":"fb73d373-f92d-4c73-bb27-61e4192eb39b","Type":"ContainerStarted","Data":"fc2f5b1b60aae5a66ca4b5e949c9e6a41f69c978bad114686abf8ed22f86586a"}
Sep 06 19:15:41.109803 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:41.109389    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-jwt4v" event={"ID":"fb73d373-f92d-4c73-bb27-61e4192eb39b","Type":"ContainerStarted","Data":"261d40358b90ee980988096ae915685462e765a9c979bcc9be7f6974ae5929d0"}
Sep 06 19:15:41.124936 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:41.124889    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-jwt4v" podStartSLOduration=1.935770497 podStartE2EDuration="3.124875976s" totalImagesPullingTime="1.189105479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:15:41.123544269 +0000 UTC m=+275.304726075" watchObservedRunningTime="2026-09-06 19:15:41.124875976 +0000 UTC m=+275.306057805"
Sep 06 19:15:43.088002 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.087961    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-5vf4h"]
Sep 06 19:15:43.091754 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.091734    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5vf4h"
Sep 06 19:15:43.094239 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.094210    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 06 19:15:43.094341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.094213    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 06 19:15:43.094571 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.094554    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xw2fz\""
Sep 06 19:15:43.101488 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.101467    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-5vf4h"]
Sep 06 19:15:43.116317 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.116295    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c8h6d"]
Sep 06 19:15:43.119919 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.119902    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.122546 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.122522    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w8pjt\""
Sep 06 19:15:43.122883 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.122866    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 06 19:15:43.123060 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.123026    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 06 19:15:43.123149 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.123133    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 06 19:15:43.209088 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.209065    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-node-exporter-wtmp\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.209233 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.209095    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27cdabc3-8f8e-400d-a868-7fe028fdffe7-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-5vf4h\" (UID: \"27cdabc3-8f8e-400d-a868-7fe028fdffe7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5vf4h"
Sep 06 19:15:43.209233 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.209115    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-root\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.209233 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.209133    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-sys\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.209233 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.209153    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-node-exporter-tls\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.209233 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.209224    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/27cdabc3-8f8e-400d-a868-7fe028fdffe7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-5vf4h\" (UID: \"27cdabc3-8f8e-400d-a868-7fe028fdffe7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5vf4h"
Sep 06 19:15:43.209423 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.209250    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-node-exporter-textfile\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.209423 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.209269    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27cdabc3-8f8e-400d-a868-7fe028fdffe7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-5vf4h\" (UID: \"27cdabc3-8f8e-400d-a868-7fe028fdffe7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5vf4h"
Sep 06 19:15:43.209423 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.209305    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9skj\" (UniqueName: \"kubernetes.io/projected/27cdabc3-8f8e-400d-a868-7fe028fdffe7-kube-api-access-b9skj\") pod \"openshift-state-metrics-657fd6c54-5vf4h\" (UID: \"27cdabc3-8f8e-400d-a868-7fe028fdffe7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5vf4h"
Sep 06 19:15:43.209423 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.209346    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.209423 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.209367    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-node-exporter-accelerators-collector-config\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.209423 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.209409    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-metrics-client-ca\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.209642 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.209425    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5gpv\" (UniqueName: \"kubernetes.io/projected/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-kube-api-access-f5gpv\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.310758 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.310724    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9skj\" (UniqueName: \"kubernetes.io/projected/27cdabc3-8f8e-400d-a868-7fe028fdffe7-kube-api-access-b9skj\") pod \"openshift-state-metrics-657fd6c54-5vf4h\" (UID: \"27cdabc3-8f8e-400d-a868-7fe028fdffe7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5vf4h"
Sep 06 19:15:43.310920 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.310792    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.310920 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.310824    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-node-exporter-accelerators-collector-config\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.310920 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.310867    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-metrics-client-ca\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.310920 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.310892    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5gpv\" (UniqueName: \"kubernetes.io/projected/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-kube-api-access-f5gpv\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.311129 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.310919    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-node-exporter-wtmp\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.311129 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.310949    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27cdabc3-8f8e-400d-a868-7fe028fdffe7-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-5vf4h\" (UID: \"27cdabc3-8f8e-400d-a868-7fe028fdffe7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5vf4h"
Sep 06 19:15:43.311129 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.310972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-root\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.311129 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.310997    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-sys\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.311129 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.311020    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-node-exporter-tls\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.311129 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.311051    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/27cdabc3-8f8e-400d-a868-7fe028fdffe7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-5vf4h\" (UID: \"27cdabc3-8f8e-400d-a868-7fe028fdffe7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5vf4h"
Sep 06 19:15:43.311129 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.311076    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-node-exporter-textfile\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.311129 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.311104    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-sys\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.311129 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.311100    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-node-exporter-wtmp\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.311623 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.311188    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-root\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.311623 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.311105    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27cdabc3-8f8e-400d-a868-7fe028fdffe7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-5vf4h\" (UID: \"27cdabc3-8f8e-400d-a868-7fe028fdffe7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5vf4h"
Sep 06 19:15:43.311623 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.311519    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-node-exporter-accelerators-collector-config\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.311623 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.311523    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-metrics-client-ca\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.311828 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:43.311626    2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 06 19:15:43.311828 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:43.311702    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-node-exporter-tls podName:140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c nodeName:}" failed. No retries permitted until 2026-09-06 19:15:43.811684528 +0000 UTC m=+277.992866317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-node-exporter-tls") pod "node-exporter-c8h6d" (UID: "140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c") : secret "node-exporter-tls" not found
Sep 06 19:15:43.311828 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.311745    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-node-exporter-textfile\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.311828 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.311805    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27cdabc3-8f8e-400d-a868-7fe028fdffe7-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-5vf4h\" (UID: \"27cdabc3-8f8e-400d-a868-7fe028fdffe7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5vf4h"
Sep 06 19:15:43.313894 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.313864    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.314058 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.314040    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/27cdabc3-8f8e-400d-a868-7fe028fdffe7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-5vf4h\" (UID: \"27cdabc3-8f8e-400d-a868-7fe028fdffe7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5vf4h"
Sep 06 19:15:43.314249 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.314229    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27cdabc3-8f8e-400d-a868-7fe028fdffe7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-5vf4h\" (UID: \"27cdabc3-8f8e-400d-a868-7fe028fdffe7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5vf4h"
Sep 06 19:15:43.322787 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.322758    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9skj\" (UniqueName: \"kubernetes.io/projected/27cdabc3-8f8e-400d-a868-7fe028fdffe7-kube-api-access-b9skj\") pod \"openshift-state-metrics-657fd6c54-5vf4h\" (UID: \"27cdabc3-8f8e-400d-a868-7fe028fdffe7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5vf4h"
Sep 06 19:15:43.323424 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.323403    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5gpv\" (UniqueName: \"kubernetes.io/projected/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-kube-api-access-f5gpv\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.404003 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.403954    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5vf4h"
Sep 06 19:15:43.683380 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.683298    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-5vf4h"]
Sep 06 19:15:43.686098 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:15:43.686065    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27cdabc3_8f8e_400d_a868_7fe028fdffe7.slice/crio-a2d9d5dcbd0c5bc6a70ae5d82b97f7352c48b738733391f4e74fcbe6acb57a37 WatchSource:0}: Error finding container a2d9d5dcbd0c5bc6a70ae5d82b97f7352c48b738733391f4e74fcbe6acb57a37: Status 404 returned error can't find the container with id a2d9d5dcbd0c5bc6a70ae5d82b97f7352c48b738733391f4e74fcbe6acb57a37
Sep 06 19:15:43.815784 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.815755    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-node-exporter-tls\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:43.818058 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:43.818029    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c-node-exporter-tls\") pod \"node-exporter-c8h6d\" (UID: \"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c\") " pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:44.030131 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.030107    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c8h6d"
Sep 06 19:15:44.040926 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:15:44.040893    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod140b4a5c_ce0b_40ae_b4eb_26f17ef8d00c.slice/crio-50bf77569377f7b1786e17d4a6c61aef45146613b8f7ae4d7f185d5ee0e0cc90 WatchSource:0}: Error finding container 50bf77569377f7b1786e17d4a6c61aef45146613b8f7ae4d7f185d5ee0e0cc90: Status 404 returned error can't find the container with id 50bf77569377f7b1786e17d4a6c61aef45146613b8f7ae4d7f185d5ee0e0cc90
Sep 06 19:15:44.122347 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.122314    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5vf4h" event={"ID":"27cdabc3-8f8e-400d-a868-7fe028fdffe7","Type":"ContainerStarted","Data":"8b713ca5a8774ff389d5a451a75825a59430ecb620d928882d4d8e5e08031e7d"}
Sep 06 19:15:44.122855 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.122382    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5vf4h" event={"ID":"27cdabc3-8f8e-400d-a868-7fe028fdffe7","Type":"ContainerStarted","Data":"c6176a942c21f40301c11cd2f075bcf85a8dc17d7c6c78704ae704432d01e540"}
Sep 06 19:15:44.122855 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.122410    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5vf4h" event={"ID":"27cdabc3-8f8e-400d-a868-7fe028fdffe7","Type":"ContainerStarted","Data":"a2d9d5dcbd0c5bc6a70ae5d82b97f7352c48b738733391f4e74fcbe6acb57a37"}
Sep 06 19:15:44.123509 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.123488    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c8h6d" event={"ID":"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c","Type":"ContainerStarted","Data":"50bf77569377f7b1786e17d4a6c61aef45146613b8f7ae4d7f185d5ee0e0cc90"}
Sep 06 19:15:44.151619 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.151580    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 06 19:15:44.157421 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.157401    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.159996 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.159965    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 06 19:15:44.159996 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.159966    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 06 19:15:44.160183 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.160054    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 06 19:15:44.160183 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.160152    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 06 19:15:44.160396 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.160354    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 06 19:15:44.160396 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.160386    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 06 19:15:44.160582 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.160503    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 06 19:15:44.160674 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.160654    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 06 19:15:44.160746 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.160734    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 06 19:15:44.160977 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.160961    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-79m96\""
Sep 06 19:15:44.168036 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.168017    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 06 19:15:44.320135 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.320050    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.320135 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.320128    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/85f39080-572f-4247-9a67-4909d837d4ac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.320337 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.320156    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.320337 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.320176    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85f39080-572f-4247-9a67-4909d837d4ac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.320337 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.320202    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.320337 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.320231    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sllcc\" (UniqueName: \"kubernetes.io/projected/85f39080-572f-4247-9a67-4909d837d4ac-kube-api-access-sllcc\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.320337 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.320259    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-config-volume\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.320337 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.320280    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/85f39080-572f-4247-9a67-4909d837d4ac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.320337 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.320313    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-web-config\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.320665 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.320393    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.320665 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.320489    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85f39080-572f-4247-9a67-4909d837d4ac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.320665 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.320559    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/85f39080-572f-4247-9a67-4909d837d4ac-config-out\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.320665 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.320622    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.421413 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.421376    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-config-volume\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.421563 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.421422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/85f39080-572f-4247-9a67-4909d837d4ac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.421563 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.421454    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-web-config\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.421563 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.421481    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.421563 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.421529    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85f39080-572f-4247-9a67-4909d837d4ac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.421563 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.421562    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/85f39080-572f-4247-9a67-4909d837d4ac-config-out\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.421838 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.421587    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.421838 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.421649    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.421838 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.421699    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/85f39080-572f-4247-9a67-4909d837d4ac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.421838 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.421723    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.421838 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.421745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85f39080-572f-4247-9a67-4909d837d4ac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.421838 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.421772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.421838 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.421801    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sllcc\" (UniqueName: \"kubernetes.io/projected/85f39080-572f-4247-9a67-4909d837d4ac-kube-api-access-sllcc\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.422926 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.422372    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/85f39080-572f-4247-9a67-4909d837d4ac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.423516 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.423207    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85f39080-572f-4247-9a67-4909d837d4ac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.424570 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.424543    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-web-config\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.424721 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.424699    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-config-volume\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.425155 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.425132    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85f39080-572f-4247-9a67-4909d837d4ac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.425920 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.425877    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.426013 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.425967    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.426013 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.425996    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/85f39080-572f-4247-9a67-4909d837d4ac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.426135 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.426026    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.426234 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.426211    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/85f39080-572f-4247-9a67-4909d837d4ac-config-out\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.426389 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.426370    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.427426 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.427403    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.429058 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.429040    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sllcc\" (UniqueName: \"kubernetes.io/projected/85f39080-572f-4247-9a67-4909d837d4ac-kube-api-access-sllcc\") pod \"alertmanager-main-0\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.466876 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.466859    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:15:44.745907 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:44.745876    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 06 19:15:44.904416 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:15:44.904388    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f39080_572f_4247_9a67_4909d837d4ac.slice/crio-aa2cf83d84e68a29fc5483409ac7b8264896f9aa9aa1fb7875b7b520b73235cd WatchSource:0}: Error finding container aa2cf83d84e68a29fc5483409ac7b8264896f9aa9aa1fb7875b7b520b73235cd: Status 404 returned error can't find the container with id aa2cf83d84e68a29fc5483409ac7b8264896f9aa9aa1fb7875b7b520b73235cd
Sep 06 19:15:45.128417 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:45.128355    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"85f39080-572f-4247-9a67-4909d837d4ac","Type":"ContainerStarted","Data":"aa2cf83d84e68a29fc5483409ac7b8264896f9aa9aa1fb7875b7b520b73235cd"}
Sep 06 19:15:45.774556 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:45.774515    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-gqxrr" podUID="a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88"
Sep 06 19:15:46.134585 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:46.134511    2566 generic.go:356] "Generic (PLEG): container finished" podID="140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c" containerID="f146e4b87db2b0067e2f9ad84fb55c47eef28dc0b5593c7b52b23cd647cb44f8" exitCode=0
Sep 06 19:15:46.134986 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:46.134666    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c8h6d" event={"ID":"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c","Type":"ContainerDied","Data":"f146e4b87db2b0067e2f9ad84fb55c47eef28dc0b5593c7b52b23cd647cb44f8"}
Sep 06 19:15:46.137625 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:46.137576    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5vf4h" event={"ID":"27cdabc3-8f8e-400d-a868-7fe028fdffe7","Type":"ContainerStarted","Data":"76683376a57ef828a0bcbcb0fb4190078c0f733329bd23bf65bcc2fe5691491e"}
Sep 06 19:15:46.139317 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:46.139288    2566 generic.go:356] "Generic (PLEG): container finished" podID="85f39080-572f-4247-9a67-4909d837d4ac" containerID="ad97888faac80bb5b1e0fc7f62c95467ef833ea612aa7eb5fe581fa2d75a641c" exitCode=0
Sep 06 19:15:46.139413 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:46.139335    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"85f39080-572f-4247-9a67-4909d837d4ac","Type":"ContainerDied","Data":"ad97888faac80bb5b1e0fc7f62c95467ef833ea612aa7eb5fe581fa2d75a641c"}
Sep 06 19:15:46.139413 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:46.139385    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:15:46.169491 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:46.169451    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5vf4h" podStartSLOduration=1.8032433939999999 podStartE2EDuration="3.169435008s" totalImagesPullingTime="1.366191614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:15:46.168059856 +0000 UTC m=+280.349241664" watchObservedRunningTime="2026-09-06 19:15:46.169435008 +0000 UTC m=+280.350616835"
Sep 06 19:15:47.146219 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:47.146133    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c8h6d" event={"ID":"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c","Type":"ContainerStarted","Data":"7fb532a95adcccb3adbd108af824a81b664f5ec7cf3c67fb90eb25ff7cd53bca"}
Sep 06 19:15:47.146219 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:47.146191    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c8h6d" event={"ID":"140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c","Type":"ContainerStarted","Data":"97e2f2e10437cefecd823a72ff1b4ebfcada3c8c43ef557f4ec5a0910957b12e"}
Sep 06 19:15:47.167309 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:47.167268    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c8h6d" podStartSLOduration=2.97777212 podStartE2EDuration="4.167249568s" totalImagesPullingTime="1.121873268s" totalInitContainerRuntime="67.60418ms" isStatefulPod=true podCreationTimestamp="2026-09-06 19:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:15:47.165477878 +0000 UTC m=+281.346659683" watchObservedRunningTime="2026-09-06 19:15:47.167249568 +0000 UTC m=+281.348431373"
Sep 06 19:15:48.153437 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:48.153355    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"85f39080-572f-4247-9a67-4909d837d4ac","Type":"ContainerStarted","Data":"20a84e6ee3da537b32e9203c14d1f849521d57782c0d382355fb7acefce011fe"}
Sep 06 19:15:48.153901 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:48.153467    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"85f39080-572f-4247-9a67-4909d837d4ac","Type":"ContainerStarted","Data":"7034af99f0307c66ae5317a9d1da312b8e093ceee70dc601cf84ac1a23efadf3"}
Sep 06 19:15:48.153901 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:48.153516    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"85f39080-572f-4247-9a67-4909d837d4ac","Type":"ContainerStarted","Data":"b4a93d81eef80984cccfb8d6d15398a15e3b611cba89fddebb1e17fe33310817"}
Sep 06 19:15:48.153901 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:48.153562    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"85f39080-572f-4247-9a67-4909d837d4ac","Type":"ContainerStarted","Data":"cb45d143c9984bdd4b39d64b1bdcc46ec9508c99d08ff936e3dc92f5b7cc21ae"}
Sep 06 19:15:48.153901 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:48.153628    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"85f39080-572f-4247-9a67-4909d837d4ac","Type":"ContainerStarted","Data":"45f3cf264e25806d96c850eaf6c7b1a8714cae94145c928f706007f9449a5914"}
Sep 06 19:15:48.761963 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:48.761888    2566 patch_prober.go:28] interesting pod/image-registry-d45bfdd8d-96984 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 06 19:15:48.761963 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:48.761938    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-d45bfdd8d-96984" podUID="1f1f8330-fe91-47ea-9ea4-766dff7f934e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 06 19:15:49.102659 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:49.102555    2566 patch_prober.go:28] interesting pod/image-registry-6dcf6d58c-nd6k8 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 06 19:15:49.102659 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:49.102637    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8" podUID="22089faf-55a4-4003-849a-56580918f07b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 06 19:15:49.162540 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:49.162478    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"85f39080-572f-4247-9a67-4909d837d4ac","Type":"ContainerStarted","Data":"47279329b5138f77b1d7c9fc2543a5ba72d24aadb8b4aaf758009e61917968d2"}
Sep 06 19:15:49.187904 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:49.187853    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.119721974 podStartE2EDuration="5.187829191s" totalImagesPullingTime="3.052366344s" totalInitContainerRuntime="15.740873ms" isStatefulPod=false podCreationTimestamp="2026-09-06 19:15:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:15:49.185784816 +0000 UTC m=+283.366966633" watchObservedRunningTime="2026-09-06 19:15:49.187829191 +0000 UTC m=+283.369010996"
Sep 06 19:15:49.260181 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:49.259721    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls\") pod \"dns-default-gqxrr\" (UID: \"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88\") " pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:15:49.263350 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:49.263323    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88-metrics-tls\") pod \"dns-default-gqxrr\" (UID: \"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88\") " pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:15:49.442362 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:49.442277    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nxfsd\""
Sep 06 19:15:49.451286 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:49.451257    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:15:49.728982 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:49.728944    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gqxrr"]
Sep 06 19:15:49.731563 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:15:49.731537    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2376ae5_11e6_4a7f_9e5d_9f75a0c43f88.slice/crio-766f0378d13c51746ad32fb2828b6b96f349a5389a68d41de966485cfe804aa9 WatchSource:0}: Error finding container 766f0378d13c51746ad32fb2828b6b96f349a5389a68d41de966485cfe804aa9: Status 404 returned error can't find the container with id 766f0378d13c51746ad32fb2828b6b96f349a5389a68d41de966485cfe804aa9
Sep 06 19:15:50.168451 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:50.168366    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gqxrr" event={"ID":"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88","Type":"ContainerStarted","Data":"766f0378d13c51746ad32fb2828b6b96f349a5389a68d41de966485cfe804aa9"}
Sep 06 19:15:51.073150 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:51.073110    2566 patch_prober.go:28] interesting pod/image-registry-6dcf6d58c-nd6k8 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 06 19:15:51.073303 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:51.073173    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8" podUID="22089faf-55a4-4003-849a-56580918f07b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 06 19:15:52.179810 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:52.179775    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gqxrr" event={"ID":"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88","Type":"ContainerStarted","Data":"f02f77671181411d0797755a3933cdf0d49f6442b0414b1062d9868fec693543"}
Sep 06 19:15:52.180196 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:52.179862    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gqxrr" event={"ID":"a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88","Type":"ContainerStarted","Data":"ea869628d4ff2afa6d05d51ed54762bb27693f1e79637d9760ccdb29e6afaa12"}
Sep 06 19:15:52.180196 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:52.179919    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:15:52.197369 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:52.197325    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gqxrr" podStartSLOduration=251.610022288 podStartE2EDuration="4m13.197313392s" totalImagesPullingTime="1.587291104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:11:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:15:52.196452583 +0000 UTC m=+286.377634393" watchObservedRunningTime="2026-09-06 19:15:52.197313392 +0000 UTC m=+286.378495198"
Sep 06 19:15:53.021309 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.021278    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65bcfcb85b-4qk6r"]
Sep 06 19:15:53.024565 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.024550    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:15:53.027883 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.027859    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nb98g\""
Sep 06 19:15:53.027883 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.027859    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 06 19:15:53.028131 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.027858    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 06 19:15:53.028131 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.027972    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 06 19:15:53.028131 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.028083    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 06 19:15:53.028260 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.028182    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 06 19:15:53.028260 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.028182    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 06 19:15:53.028400 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.028385    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 06 19:15:53.032386 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.032370    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 06 19:15:53.034374 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.034356    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65bcfcb85b-4qk6r"]
Sep 06 19:15:53.195303 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.195278    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b93bde3b-ed23-4b7f-b04e-9a80b1883382-console-serving-cert\") pod \"console-65bcfcb85b-4qk6r\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") " pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:15:53.195682 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.195319    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tv7w\" (UniqueName: \"kubernetes.io/projected/b93bde3b-ed23-4b7f-b04e-9a80b1883382-kube-api-access-8tv7w\") pod \"console-65bcfcb85b-4qk6r\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") " pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:15:53.195682 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.195389    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-console-config\") pod \"console-65bcfcb85b-4qk6r\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") " pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:15:53.195682 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.195422    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b93bde3b-ed23-4b7f-b04e-9a80b1883382-console-oauth-config\") pod \"console-65bcfcb85b-4qk6r\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") " pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:15:53.195682 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.195470    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-trusted-ca-bundle\") pod \"console-65bcfcb85b-4qk6r\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") " pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:15:53.195682 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.195501    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-oauth-serving-cert\") pod \"console-65bcfcb85b-4qk6r\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") " pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:15:53.195682 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.195556    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-service-ca\") pod \"console-65bcfcb85b-4qk6r\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") " pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:15:53.296126 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.296063    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-console-config\") pod \"console-65bcfcb85b-4qk6r\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") " pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:15:53.296126 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.296119    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b93bde3b-ed23-4b7f-b04e-9a80b1883382-console-oauth-config\") pod \"console-65bcfcb85b-4qk6r\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") " pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:15:53.296297 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.296135    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-trusted-ca-bundle\") pod \"console-65bcfcb85b-4qk6r\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") " pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:15:53.296297 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.296279    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-oauth-serving-cert\") pod \"console-65bcfcb85b-4qk6r\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") " pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:15:53.296395 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.296353    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-service-ca\") pod \"console-65bcfcb85b-4qk6r\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") " pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:15:53.296455 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.296400    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b93bde3b-ed23-4b7f-b04e-9a80b1883382-console-serving-cert\") pod \"console-65bcfcb85b-4qk6r\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") " pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:15:53.296507 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.296452    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tv7w\" (UniqueName: \"kubernetes.io/projected/b93bde3b-ed23-4b7f-b04e-9a80b1883382-kube-api-access-8tv7w\") pod \"console-65bcfcb85b-4qk6r\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") " pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:15:53.296933 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.296899    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-console-config\") pod \"console-65bcfcb85b-4qk6r\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") " pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:15:53.297201 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.296999    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-trusted-ca-bundle\") pod \"console-65bcfcb85b-4qk6r\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") " pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:15:53.297201 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.297040    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-service-ca\") pod \"console-65bcfcb85b-4qk6r\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") " pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:15:53.297201 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.297048    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-oauth-serving-cert\") pod \"console-65bcfcb85b-4qk6r\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") " pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:15:53.298621 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.298578    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b93bde3b-ed23-4b7f-b04e-9a80b1883382-console-oauth-config\") pod \"console-65bcfcb85b-4qk6r\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") " pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:15:53.298923 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.298904    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b93bde3b-ed23-4b7f-b04e-9a80b1883382-console-serving-cert\") pod \"console-65bcfcb85b-4qk6r\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") " pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:15:53.304522 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.304503    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tv7w\" (UniqueName: \"kubernetes.io/projected/b93bde3b-ed23-4b7f-b04e-9a80b1883382-kube-api-access-8tv7w\") pod \"console-65bcfcb85b-4qk6r\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") " pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:15:53.334517 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.334492    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:15:53.621172 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.621149    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65bcfcb85b-4qk6r"]
Sep 06 19:15:53.623071 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:15:53.623035    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93bde3b_ed23_4b7f_b04e_9a80b1883382.slice/crio-f81552dc1b3155d7bdb2b4d5c25fbae12bff2f6c033b29d95dab520ffc7c692e WatchSource:0}: Error finding container f81552dc1b3155d7bdb2b4d5c25fbae12bff2f6c033b29d95dab520ffc7c692e: Status 404 returned error can't find the container with id f81552dc1b3155d7bdb2b4d5c25fbae12bff2f6c033b29d95dab520ffc7c692e
Sep 06 19:15:53.776496 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.776458    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-d45bfdd8d-96984" podUID="1f1f8330-fe91-47ea-9ea4-766dff7f934e" containerName="registry" containerID="cri-o://d923390e905122cc5bf66c220db53d5fe181712a72cc98bb704a0bd05ad11e7d" gracePeriod=30
Sep 06 19:15:53.997239 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:53.997217    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:54.103257 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.103221    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f1f8330-fe91-47ea-9ea4-766dff7f934e-trusted-ca\") pod \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") "
Sep 06 19:15:54.103413 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.103275    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqz8v\" (UniqueName: \"kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-kube-api-access-sqz8v\") pod \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") "
Sep 06 19:15:54.103413 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.103310    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1f1f8330-fe91-47ea-9ea4-766dff7f934e-image-registry-private-configuration\") pod \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") "
Sep 06 19:15:54.105529 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.103732    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-bound-sa-token\") pod \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") "
Sep 06 19:15:54.105529 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.103787    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1f1f8330-fe91-47ea-9ea4-766dff7f934e-ca-trust-extracted\") pod \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") "
Sep 06 19:15:54.105529 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.103812    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-tls\") pod \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") "
Sep 06 19:15:54.105529 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.103856    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1f1f8330-fe91-47ea-9ea4-766dff7f934e-installation-pull-secrets\") pod \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") "
Sep 06 19:15:54.105529 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.103926    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-certificates\") pod \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\" (UID: \"1f1f8330-fe91-47ea-9ea4-766dff7f934e\") "
Sep 06 19:15:54.105529 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.104444    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1f1f8330-fe91-47ea-9ea4-766dff7f934e" (UID: "1f1f8330-fe91-47ea-9ea4-766dff7f934e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:15:54.105529 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.105341    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1f8330-fe91-47ea-9ea4-766dff7f934e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1f1f8330-fe91-47ea-9ea4-766dff7f934e" (UID: "1f1f8330-fe91-47ea-9ea4-766dff7f934e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:15:54.108510 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.108473    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-kube-api-access-sqz8v" (OuterVolumeSpecName: "kube-api-access-sqz8v") pod "1f1f8330-fe91-47ea-9ea4-766dff7f934e" (UID: "1f1f8330-fe91-47ea-9ea4-766dff7f934e"). InnerVolumeSpecName "kube-api-access-sqz8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:15:54.108833 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.108802    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1f1f8330-fe91-47ea-9ea4-766dff7f934e" (UID: "1f1f8330-fe91-47ea-9ea4-766dff7f934e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:15:54.109233 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.108891    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f1f8330-fe91-47ea-9ea4-766dff7f934e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1f1f8330-fe91-47ea-9ea4-766dff7f934e" (UID: "1f1f8330-fe91-47ea-9ea4-766dff7f934e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:15:54.109233 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.109115    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1f1f8330-fe91-47ea-9ea4-766dff7f934e" (UID: "1f1f8330-fe91-47ea-9ea4-766dff7f934e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:15:54.109411 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.109391    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f1f8330-fe91-47ea-9ea4-766dff7f934e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1f1f8330-fe91-47ea-9ea4-766dff7f934e" (UID: "1f1f8330-fe91-47ea-9ea4-766dff7f934e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:15:54.115525 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.115449    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1f8330-fe91-47ea-9ea4-766dff7f934e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1f1f8330-fe91-47ea-9ea4-766dff7f934e" (UID: "1f1f8330-fe91-47ea-9ea4-766dff7f934e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:15:54.190892 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.190774    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65bcfcb85b-4qk6r" event={"ID":"b93bde3b-ed23-4b7f-b04e-9a80b1883382","Type":"ContainerStarted","Data":"f81552dc1b3155d7bdb2b4d5c25fbae12bff2f6c033b29d95dab520ffc7c692e"}
Sep 06 19:15:54.192215 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.192185    2566 generic.go:356] "Generic (PLEG): container finished" podID="1f1f8330-fe91-47ea-9ea4-766dff7f934e" containerID="d923390e905122cc5bf66c220db53d5fe181712a72cc98bb704a0bd05ad11e7d" exitCode=0
Sep 06 19:15:54.192326 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.192239    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d45bfdd8d-96984"
Sep 06 19:15:54.192326 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.192244    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d45bfdd8d-96984" event={"ID":"1f1f8330-fe91-47ea-9ea4-766dff7f934e","Type":"ContainerDied","Data":"d923390e905122cc5bf66c220db53d5fe181712a72cc98bb704a0bd05ad11e7d"}
Sep 06 19:15:54.192326 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.192289    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d45bfdd8d-96984" event={"ID":"1f1f8330-fe91-47ea-9ea4-766dff7f934e","Type":"ContainerDied","Data":"2d065a787b58e2b2d29db58f82f03dc3fd0ce58b121da7a7d966ff3b736b4f04"}
Sep 06 19:15:54.192326 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.192314    2566 scope.go:117] "RemoveContainer" containerID="d923390e905122cc5bf66c220db53d5fe181712a72cc98bb704a0bd05ad11e7d"
Sep 06 19:15:54.205738 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.205686    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1f1f8330-fe91-47ea-9ea4-766dff7f934e-ca-trust-extracted\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:15:54.205738 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.205733    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-tls\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:15:54.206126 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.205748    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1f1f8330-fe91-47ea-9ea4-766dff7f934e-installation-pull-secrets\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:15:54.206126 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.205799    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1f1f8330-fe91-47ea-9ea4-766dff7f934e-registry-certificates\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:15:54.206126 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.205816    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f1f8330-fe91-47ea-9ea4-766dff7f934e-trusted-ca\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:15:54.206126 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.205829    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqz8v\" (UniqueName: \"kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-kube-api-access-sqz8v\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:15:54.206126 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.205840    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1f1f8330-fe91-47ea-9ea4-766dff7f934e-image-registry-private-configuration\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:15:54.206126 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.205855    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f1f8330-fe91-47ea-9ea4-766dff7f934e-bound-sa-token\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:15:54.206825 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.206807    2566 scope.go:117] "RemoveContainer" containerID="d923390e905122cc5bf66c220db53d5fe181712a72cc98bb704a0bd05ad11e7d"
Sep 06 19:15:54.207166 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:54.207140    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d923390e905122cc5bf66c220db53d5fe181712a72cc98bb704a0bd05ad11e7d\": container with ID starting with d923390e905122cc5bf66c220db53d5fe181712a72cc98bb704a0bd05ad11e7d not found: ID does not exist" containerID="d923390e905122cc5bf66c220db53d5fe181712a72cc98bb704a0bd05ad11e7d"
Sep 06 19:15:54.207249 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.207176    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d923390e905122cc5bf66c220db53d5fe181712a72cc98bb704a0bd05ad11e7d"} err="failed to get container status \"d923390e905122cc5bf66c220db53d5fe181712a72cc98bb704a0bd05ad11e7d\": rpc error: code = NotFound desc = could not find container \"d923390e905122cc5bf66c220db53d5fe181712a72cc98bb704a0bd05ad11e7d\": container with ID starting with d923390e905122cc5bf66c220db53d5fe181712a72cc98bb704a0bd05ad11e7d not found: ID does not exist"
Sep 06 19:15:54.216078 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.216031    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d45bfdd8d-96984"]
Sep 06 19:15:54.219247 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.219225    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-d45bfdd8d-96984"]
Sep 06 19:15:54.313555 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:54.313233    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f1f8330-fe91-47ea-9ea4-766dff7f934e" path="/var/lib/kubelet/pods/1f1f8330-fe91-47ea-9ea4-766dff7f934e/volumes"
Sep 06 19:15:57.208993 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:57.208952    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65bcfcb85b-4qk6r" event={"ID":"b93bde3b-ed23-4b7f-b04e-9a80b1883382","Type":"ContainerStarted","Data":"a1aed2f3d84e72afeabf94e5365d4dfccb562acf6a9b051981aa3d0101271957"}
Sep 06 19:15:57.226288 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:57.226205    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65bcfcb85b-4qk6r" podStartSLOduration=1.390310059 podStartE2EDuration="4.226187375s" totalImagesPullingTime="2.835877316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:15:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:15:57.225816618 +0000 UTC m=+291.406998424" watchObservedRunningTime="2026-09-06 19:15:57.226187375 +0000 UTC m=+291.407369178"
Sep 06 19:15:57.304843 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:15:57.304807    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-qlvsw" podUID="83a7379b-ab58-4439-998e-448e48150f3b"
Sep 06 19:15:59.101957 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:59.101922    2566 patch_prober.go:28] interesting pod/image-registry-6dcf6d58c-nd6k8 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 06 19:15:59.102384 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:15:59.101974    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8" podUID="22089faf-55a4-4003-849a-56580918f07b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 06 19:16:01.072251 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:16:01.072221    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6dcf6d58c-nd6k8"
Sep 06 19:16:02.186705 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:16:02.186675    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gqxrr"
Sep 06 19:16:03.334789 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:16:03.334756    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:16:03.335210 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:16:03.334806    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:16:03.340569 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:16:03.340548    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:16:04.239619 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:16:04.239578    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:16:06.261111 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:16:06.261086    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnssq_9254e6f1-b73d-49cc-8bc1-513925ccde4f/ovn-acl-logging/0.log"
Sep 06 19:16:06.261465 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:16:06.261163    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnssq_9254e6f1-b73d-49cc-8bc1-513925ccde4f/ovn-acl-logging/0.log"
Sep 06 19:16:06.267087 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:16:06.267068    2566 kubelet.go:1648] "Image garbage collection succeeded"
Sep 06 19:16:12.303956 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:16:12.303926    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:16:33.343099 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:16:33.343013    2566 generic.go:356] "Generic (PLEG): container finished" podID="033d0561-8f3c-4e3c-afad-88da2108830e" containerID="dfa69f95e3f1e28e5157e426a3643c9b1c018b150e61f0b83a074ff092ab0f37" exitCode=0
Sep 06 19:16:33.343099 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:16:33.343061    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4" event={"ID":"033d0561-8f3c-4e3c-afad-88da2108830e","Type":"ContainerDied","Data":"dfa69f95e3f1e28e5157e426a3643c9b1c018b150e61f0b83a074ff092ab0f37"}
Sep 06 19:16:33.343584 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:16:33.343403    2566 scope.go:117] "RemoveContainer" containerID="dfa69f95e3f1e28e5157e426a3643c9b1c018b150e61f0b83a074ff092ab0f37"
Sep 06 19:16:34.348388 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:16:34.348356    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-djlz4" event={"ID":"033d0561-8f3c-4e3c-afad-88da2108830e","Type":"ContainerStarted","Data":"b36ce1a66e04c8d1952d85417c995a98eab19f3d814b76eda7932fab16645be4"}
Sep 06 19:17:03.320745 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:03.320652    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 06 19:17:03.321250 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:03.321034    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="alertmanager" containerID="cri-o://45f3cf264e25806d96c850eaf6c7b1a8714cae94145c928f706007f9449a5914" gracePeriod=120
Sep 06 19:17:03.321250 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:03.321084    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="kube-rbac-proxy-metric" containerID="cri-o://20a84e6ee3da537b32e9203c14d1f849521d57782c0d382355fb7acefce011fe" gracePeriod=120
Sep 06 19:17:03.321250 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:03.321122    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="prom-label-proxy" containerID="cri-o://47279329b5138f77b1d7c9fc2543a5ba72d24aadb8b4aaf758009e61917968d2" gracePeriod=120
Sep 06 19:17:03.321250 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:03.321136    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="kube-rbac-proxy-web" containerID="cri-o://b4a93d81eef80984cccfb8d6d15398a15e3b611cba89fddebb1e17fe33310817" gracePeriod=120
Sep 06 19:17:03.321250 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:03.321202    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="config-reloader" containerID="cri-o://cb45d143c9984bdd4b39d64b1bdcc46ec9508c99d08ff936e3dc92f5b7cc21ae" gracePeriod=120
Sep 06 19:17:03.321250 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:03.321144    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="kube-rbac-proxy" containerID="cri-o://7034af99f0307c66ae5317a9d1da312b8e093ceee70dc601cf84ac1a23efadf3" gracePeriod=120
Sep 06 19:17:03.457419 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:03.457394    2566 generic.go:356] "Generic (PLEG): container finished" podID="85f39080-572f-4247-9a67-4909d837d4ac" containerID="47279329b5138f77b1d7c9fc2543a5ba72d24aadb8b4aaf758009e61917968d2" exitCode=0
Sep 06 19:17:03.457419 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:03.457418    2566 generic.go:356] "Generic (PLEG): container finished" podID="85f39080-572f-4247-9a67-4909d837d4ac" containerID="7034af99f0307c66ae5317a9d1da312b8e093ceee70dc601cf84ac1a23efadf3" exitCode=0
Sep 06 19:17:03.457545 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:03.457424    2566 generic.go:356] "Generic (PLEG): container finished" podID="85f39080-572f-4247-9a67-4909d837d4ac" containerID="cb45d143c9984bdd4b39d64b1bdcc46ec9508c99d08ff936e3dc92f5b7cc21ae" exitCode=0
Sep 06 19:17:03.457545 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:03.457430    2566 generic.go:356] "Generic (PLEG): container finished" podID="85f39080-572f-4247-9a67-4909d837d4ac" containerID="45f3cf264e25806d96c850eaf6c7b1a8714cae94145c928f706007f9449a5914" exitCode=0
Sep 06 19:17:03.457545 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:03.457459    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"85f39080-572f-4247-9a67-4909d837d4ac","Type":"ContainerDied","Data":"47279329b5138f77b1d7c9fc2543a5ba72d24aadb8b4aaf758009e61917968d2"}
Sep 06 19:17:03.457545 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:03.457518    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"85f39080-572f-4247-9a67-4909d837d4ac","Type":"ContainerDied","Data":"7034af99f0307c66ae5317a9d1da312b8e093ceee70dc601cf84ac1a23efadf3"}
Sep 06 19:17:03.457545 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:03.457543    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"85f39080-572f-4247-9a67-4909d837d4ac","Type":"ContainerDied","Data":"cb45d143c9984bdd4b39d64b1bdcc46ec9508c99d08ff936e3dc92f5b7cc21ae"}
Sep 06 19:17:03.457726 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:03.457572    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"85f39080-572f-4247-9a67-4909d837d4ac","Type":"ContainerDied","Data":"45f3cf264e25806d96c850eaf6c7b1a8714cae94145c928f706007f9449a5914"}
Sep 06 19:17:04.465724 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.465678    2566 generic.go:356] "Generic (PLEG): container finished" podID="85f39080-572f-4247-9a67-4909d837d4ac" containerID="20a84e6ee3da537b32e9203c14d1f849521d57782c0d382355fb7acefce011fe" exitCode=0
Sep 06 19:17:04.466185 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.466168    2566 generic.go:356] "Generic (PLEG): container finished" podID="85f39080-572f-4247-9a67-4909d837d4ac" containerID="b4a93d81eef80984cccfb8d6d15398a15e3b611cba89fddebb1e17fe33310817" exitCode=0
Sep 06 19:17:04.466484 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.465727    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"85f39080-572f-4247-9a67-4909d837d4ac","Type":"ContainerDied","Data":"20a84e6ee3da537b32e9203c14d1f849521d57782c0d382355fb7acefce011fe"}
Sep 06 19:17:04.466484 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.466432    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"85f39080-572f-4247-9a67-4909d837d4ac","Type":"ContainerDied","Data":"b4a93d81eef80984cccfb8d6d15398a15e3b611cba89fddebb1e17fe33310817"}
Sep 06 19:17:04.559244 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.559220    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:04.646700 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.646598    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"85f39080-572f-4247-9a67-4909d837d4ac\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") "
Sep 06 19:17:04.646700 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.646674    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-config-volume\") pod \"85f39080-572f-4247-9a67-4909d837d4ac\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") "
Sep 06 19:17:04.646893 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.646706    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sllcc\" (UniqueName: \"kubernetes.io/projected/85f39080-572f-4247-9a67-4909d837d4ac-kube-api-access-sllcc\") pod \"85f39080-572f-4247-9a67-4909d837d4ac\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") "
Sep 06 19:17:04.646893 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.646737    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-main-tls\") pod \"85f39080-572f-4247-9a67-4909d837d4ac\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") "
Sep 06 19:17:04.646893 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.646766    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-kube-rbac-proxy-web\") pod \"85f39080-572f-4247-9a67-4909d837d4ac\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") "
Sep 06 19:17:04.646893 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.646789    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-cluster-tls-config\") pod \"85f39080-572f-4247-9a67-4909d837d4ac\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") "
Sep 06 19:17:04.646893 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.646828    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85f39080-572f-4247-9a67-4909d837d4ac-metrics-client-ca\") pod \"85f39080-572f-4247-9a67-4909d837d4ac\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") "
Sep 06 19:17:04.647214 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.647186    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f39080-572f-4247-9a67-4909d837d4ac-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "85f39080-572f-4247-9a67-4909d837d4ac" (UID: "85f39080-572f-4247-9a67-4909d837d4ac"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:17:04.647653 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.647207    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/85f39080-572f-4247-9a67-4909d837d4ac-alertmanager-main-db\") pod \"85f39080-572f-4247-9a67-4909d837d4ac\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") "
Sep 06 19:17:04.647827 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.647671    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f39080-572f-4247-9a67-4909d837d4ac-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "85f39080-572f-4247-9a67-4909d837d4ac" (UID: "85f39080-572f-4247-9a67-4909d837d4ac"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:17:04.647938 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.647917    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/85f39080-572f-4247-9a67-4909d837d4ac-config-out\") pod \"85f39080-572f-4247-9a67-4909d837d4ac\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") "
Sep 06 19:17:04.648006 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.647965    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-web-config\") pod \"85f39080-572f-4247-9a67-4909d837d4ac\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") "
Sep 06 19:17:04.648006 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.648000    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/85f39080-572f-4247-9a67-4909d837d4ac-tls-assets\") pod \"85f39080-572f-4247-9a67-4909d837d4ac\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") "
Sep 06 19:17:04.648106 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.648042    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-kube-rbac-proxy\") pod \"85f39080-572f-4247-9a67-4909d837d4ac\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") "
Sep 06 19:17:04.648106 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.648078    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85f39080-572f-4247-9a67-4909d837d4ac-alertmanager-trusted-ca-bundle\") pod \"85f39080-572f-4247-9a67-4909d837d4ac\" (UID: \"85f39080-572f-4247-9a67-4909d837d4ac\") "
Sep 06 19:17:04.648364 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.648343    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85f39080-572f-4247-9a67-4909d837d4ac-metrics-client-ca\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:17:04.648443 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.648371    2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/85f39080-572f-4247-9a67-4909d837d4ac-alertmanager-main-db\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:17:04.650310 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.649663    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f39080-572f-4247-9a67-4909d837d4ac-kube-api-access-sllcc" (OuterVolumeSpecName: "kube-api-access-sllcc") pod "85f39080-572f-4247-9a67-4909d837d4ac" (UID: "85f39080-572f-4247-9a67-4909d837d4ac"). InnerVolumeSpecName "kube-api-access-sllcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:17:04.650310 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.649748    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-config-volume" (OuterVolumeSpecName: "config-volume") pod "85f39080-572f-4247-9a67-4909d837d4ac" (UID: "85f39080-572f-4247-9a67-4909d837d4ac"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:17:04.650310 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.650146    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f39080-572f-4247-9a67-4909d837d4ac-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "85f39080-572f-4247-9a67-4909d837d4ac" (UID: "85f39080-572f-4247-9a67-4909d837d4ac"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:17:04.650510 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.650438    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "85f39080-572f-4247-9a67-4909d837d4ac" (UID: "85f39080-572f-4247-9a67-4909d837d4ac"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:17:04.650510 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.650498    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "85f39080-572f-4247-9a67-4909d837d4ac" (UID: "85f39080-572f-4247-9a67-4909d837d4ac"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:17:04.650743 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.650723    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "85f39080-572f-4247-9a67-4909d837d4ac" (UID: "85f39080-572f-4247-9a67-4909d837d4ac"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:17:04.651043 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.651024    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f39080-572f-4247-9a67-4909d837d4ac-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "85f39080-572f-4247-9a67-4909d837d4ac" (UID: "85f39080-572f-4247-9a67-4909d837d4ac"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:17:04.651288 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.651270    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f39080-572f-4247-9a67-4909d837d4ac-config-out" (OuterVolumeSpecName: "config-out") pod "85f39080-572f-4247-9a67-4909d837d4ac" (UID: "85f39080-572f-4247-9a67-4909d837d4ac"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:17:04.651751 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.651729    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "85f39080-572f-4247-9a67-4909d837d4ac" (UID: "85f39080-572f-4247-9a67-4909d837d4ac"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:17:04.655311 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.655289    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "85f39080-572f-4247-9a67-4909d837d4ac" (UID: "85f39080-572f-4247-9a67-4909d837d4ac"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:17:04.661099 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.661075    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-web-config" (OuterVolumeSpecName: "web-config") pod "85f39080-572f-4247-9a67-4909d837d4ac" (UID: "85f39080-572f-4247-9a67-4909d837d4ac"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:17:04.749531 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.749500    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sllcc\" (UniqueName: \"kubernetes.io/projected/85f39080-572f-4247-9a67-4909d837d4ac-kube-api-access-sllcc\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:17:04.749531 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.749525    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-main-tls\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:17:04.749531 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.749535    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:17:04.749744 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.749545    2566 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-cluster-tls-config\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:17:04.749744 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.749554    2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/85f39080-572f-4247-9a67-4909d837d4ac-config-out\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:17:04.749744 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.749562    2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-web-config\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:17:04.749744 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.749570    2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/85f39080-572f-4247-9a67-4909d837d4ac-tls-assets\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:17:04.749744 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.749577    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:17:04.749744 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.749586    2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85f39080-572f-4247-9a67-4909d837d4ac-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:17:04.749744 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.749594    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:17:04.749744 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:04.749617    2566 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/85f39080-572f-4247-9a67-4909d837d4ac-config-volume\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:17:05.473033 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.473000    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"85f39080-572f-4247-9a67-4909d837d4ac","Type":"ContainerDied","Data":"aa2cf83d84e68a29fc5483409ac7b8264896f9aa9aa1fb7875b7b520b73235cd"}
Sep 06 19:17:05.473415 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.473044    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.473415 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.473073    2566 scope.go:117] "RemoveContainer" containerID="47279329b5138f77b1d7c9fc2543a5ba72d24aadb8b4aaf758009e61917968d2"
Sep 06 19:17:05.480482 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.480370    2566 scope.go:117] "RemoveContainer" containerID="20a84e6ee3da537b32e9203c14d1f849521d57782c0d382355fb7acefce011fe"
Sep 06 19:17:05.487015 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.486998    2566 scope.go:117] "RemoveContainer" containerID="7034af99f0307c66ae5317a9d1da312b8e093ceee70dc601cf84ac1a23efadf3"
Sep 06 19:17:05.494748 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.494198    2566 scope.go:117] "RemoveContainer" containerID="b4a93d81eef80984cccfb8d6d15398a15e3b611cba89fddebb1e17fe33310817"
Sep 06 19:17:05.495500 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.495483    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 06 19:17:05.499495 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.499469    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 06 19:17:05.502065 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.502049    2566 scope.go:117] "RemoveContainer" containerID="cb45d143c9984bdd4b39d64b1bdcc46ec9508c99d08ff936e3dc92f5b7cc21ae"
Sep 06 19:17:05.508769 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.508748    2566 scope.go:117] "RemoveContainer" containerID="45f3cf264e25806d96c850eaf6c7b1a8714cae94145c928f706007f9449a5914"
Sep 06 19:17:05.515517 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.515489    2566 scope.go:117] "RemoveContainer" containerID="ad97888faac80bb5b1e0fc7f62c95467ef833ea612aa7eb5fe581fa2d75a641c"
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525320    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525726    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="kube-rbac-proxy-web"
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525743    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="kube-rbac-proxy-web"
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525755    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="init-config-reloader"
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525763    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="init-config-reloader"
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525775    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="config-reloader"
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525782    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="config-reloader"
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525802    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="prom-label-proxy"
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525811    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="prom-label-proxy"
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525823    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f1f8330-fe91-47ea-9ea4-766dff7f934e" containerName="registry"
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525830    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f1f8330-fe91-47ea-9ea4-766dff7f934e" containerName="registry"
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525842    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="kube-rbac-proxy"
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525850    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="kube-rbac-proxy"
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525859    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="kube-rbac-proxy-metric"
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525867    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="kube-rbac-proxy-metric"
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525877    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="alertmanager"
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525884    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="alertmanager"
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525944    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="kube-rbac-proxy-web"
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525954    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="alertmanager"
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525961    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="prom-label-proxy"
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525969    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="kube-rbac-proxy"
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525976    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f1f8330-fe91-47ea-9ea4-766dff7f934e" containerName="registry"
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525984    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="config-reloader"
Sep 06 19:17:05.526772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.525991    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="85f39080-572f-4247-9a67-4909d837d4ac" containerName="kube-rbac-proxy-metric"
Sep 06 19:17:05.531056 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.531040    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.533516 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.533496    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 06 19:17:05.533647 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.533629    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 06 19:17:05.533731 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.533646    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 06 19:17:05.533786 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.533729    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 06 19:17:05.534028 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.533991    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 06 19:17:05.534130 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.534042    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 06 19:17:05.534130 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.534007    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-79m96\""
Sep 06 19:17:05.534130 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.534063    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 06 19:17:05.534391 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.534376    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 06 19:17:05.538454 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.538435    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 06 19:17:05.542927 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.542907    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 06 19:17:05.556790 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.556770    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a730856-a0dc-4cfe-895f-a973ac07f2db-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.556883 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.556799    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a730856-a0dc-4cfe-895f-a973ac07f2db-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.556883 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.556817    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a730856-a0dc-4cfe-895f-a973ac07f2db-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.556883 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.556842    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a730856-a0dc-4cfe-895f-a973ac07f2db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.557014 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.556885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a730856-a0dc-4cfe-895f-a973ac07f2db-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.557014 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.556957    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a730856-a0dc-4cfe-895f-a973ac07f2db-config-out\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.557014 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.557007    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a730856-a0dc-4cfe-895f-a973ac07f2db-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.557126 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.557026    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a730856-a0dc-4cfe-895f-a973ac07f2db-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.557126 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.557043    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a730856-a0dc-4cfe-895f-a973ac07f2db-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.557126 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.557061    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a730856-a0dc-4cfe-895f-a973ac07f2db-config-volume\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.557221 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.557120    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a730856-a0dc-4cfe-895f-a973ac07f2db-web-config\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.557221 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.557148    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a730856-a0dc-4cfe-895f-a973ac07f2db-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.557221 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.557169    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-666vg\" (UniqueName: \"kubernetes.io/projected/5a730856-a0dc-4cfe-895f-a973ac07f2db-kube-api-access-666vg\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.658319 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.658242    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a730856-a0dc-4cfe-895f-a973ac07f2db-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.658319 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.658279    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a730856-a0dc-4cfe-895f-a973ac07f2db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.658319 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.658301    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a730856-a0dc-4cfe-895f-a973ac07f2db-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.658523 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.658485    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a730856-a0dc-4cfe-895f-a973ac07f2db-config-out\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.658628 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.658583    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a730856-a0dc-4cfe-895f-a973ac07f2db-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.658698 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.658641    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a730856-a0dc-4cfe-895f-a973ac07f2db-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.658698 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.658676    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a730856-a0dc-4cfe-895f-a973ac07f2db-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.658793 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.658704    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a730856-a0dc-4cfe-895f-a973ac07f2db-config-volume\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.658868 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.658737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a730856-a0dc-4cfe-895f-a973ac07f2db-web-config\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.658932 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.658884    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a730856-a0dc-4cfe-895f-a973ac07f2db-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.658932 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.658922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-666vg\" (UniqueName: \"kubernetes.io/projected/5a730856-a0dc-4cfe-895f-a973ac07f2db-kube-api-access-666vg\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.659030 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.658960    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a730856-a0dc-4cfe-895f-a973ac07f2db-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.659030 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.658996    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a730856-a0dc-4cfe-895f-a973ac07f2db-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.659472 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.659448    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a730856-a0dc-4cfe-895f-a973ac07f2db-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.659672 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.659651    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a730856-a0dc-4cfe-895f-a973ac07f2db-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.661016 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.660390    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a730856-a0dc-4cfe-895f-a973ac07f2db-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.661493 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.661467    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a730856-a0dc-4cfe-895f-a973ac07f2db-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.661710 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.661687    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a730856-a0dc-4cfe-895f-a973ac07f2db-config-out\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.661798 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.661698    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a730856-a0dc-4cfe-895f-a973ac07f2db-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.661861 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.661840    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a730856-a0dc-4cfe-895f-a973ac07f2db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.662278 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.662241    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a730856-a0dc-4cfe-895f-a973ac07f2db-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.662434 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.662415    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a730856-a0dc-4cfe-895f-a973ac07f2db-config-volume\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.662627 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.662579    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a730856-a0dc-4cfe-895f-a973ac07f2db-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.663109 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.663085    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a730856-a0dc-4cfe-895f-a973ac07f2db-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.663675 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.663657    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a730856-a0dc-4cfe-895f-a973ac07f2db-web-config\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.668274 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.668257    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-666vg\" (UniqueName: \"kubernetes.io/projected/5a730856-a0dc-4cfe-895f-a973ac07f2db-kube-api-access-666vg\") pod \"alertmanager-main-0\" (UID: \"5a730856-a0dc-4cfe-895f-a973ac07f2db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:05.840511 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:05.840483    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:17:06.118235 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:06.118211    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 06 19:17:06.120776 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:17:06.120748    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a730856_a0dc_4cfe_895f_a973ac07f2db.slice/crio-6490c7cc5662ac2d63af7466f5758a734c7fb1d334589025c7ab0c5b09db43a5 WatchSource:0}: Error finding container 6490c7cc5662ac2d63af7466f5758a734c7fb1d334589025c7ab0c5b09db43a5: Status 404 returned error can't find the container with id 6490c7cc5662ac2d63af7466f5758a734c7fb1d334589025c7ab0c5b09db43a5
Sep 06 19:17:06.310454 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:06.310419    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85f39080-572f-4247-9a67-4909d837d4ac" path="/var/lib/kubelet/pods/85f39080-572f-4247-9a67-4909d837d4ac/volumes"
Sep 06 19:17:06.477886 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:06.477859    2566 generic.go:356] "Generic (PLEG): container finished" podID="5a730856-a0dc-4cfe-895f-a973ac07f2db" containerID="a7cbe33f25011a173222701457eb99ee830a966bc6228d3eba0db1be05b1109c" exitCode=0
Sep 06 19:17:06.478192 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:06.477900    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a730856-a0dc-4cfe-895f-a973ac07f2db","Type":"ContainerDied","Data":"a7cbe33f25011a173222701457eb99ee830a966bc6228d3eba0db1be05b1109c"}
Sep 06 19:17:06.478192 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:06.477954    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a730856-a0dc-4cfe-895f-a973ac07f2db","Type":"ContainerStarted","Data":"6490c7cc5662ac2d63af7466f5758a734c7fb1d334589025c7ab0c5b09db43a5"}
Sep 06 19:17:07.339984 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.339949    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-547fddd874-2z2wm"]
Sep 06 19:17:07.343377 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.343347    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.346021 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.345997    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 06 19:17:07.346162 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.346048    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 06 19:17:07.346423 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.346406    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 06 19:17:07.346522 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.346422    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-pqj5m\""
Sep 06 19:17:07.346522 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.346407    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 06 19:17:07.346522 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.346458    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 06 19:17:07.352813 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.352794    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 06 19:17:07.355886 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.355863    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-547fddd874-2z2wm"]
Sep 06 19:17:07.370841 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.370816    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-secret-telemeter-client\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.371037 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.371006    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-federate-client-tls\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.371138 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.371045    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-telemeter-client-tls\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.371138 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.371099    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-serving-certs-ca-bundle\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.371209 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.371164    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.371209 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.371197    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.371271 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.371245    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-metrics-client-ca\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.371303 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.371283    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lbpz\" (UniqueName: \"kubernetes.io/projected/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-kube-api-access-6lbpz\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.471993 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.471959    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.471993 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.471998    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.472258 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.472024    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-metrics-client-ca\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.472258 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.472059    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lbpz\" (UniqueName: \"kubernetes.io/projected/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-kube-api-access-6lbpz\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.472258 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.472141    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-secret-telemeter-client\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.472258 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.472192    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-federate-client-tls\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.472258 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.472223    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-telemeter-client-tls\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.472718 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.472691    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-serving-certs-ca-bundle\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.473173 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.473116    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-metrics-client-ca\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.473760 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.473735    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-serving-certs-ca-bundle\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.473873 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.473839    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.475671 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.475109    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.475671 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.475579    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-secret-telemeter-client\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.476643 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.476594    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-federate-client-tls\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.476722 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.476681    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-telemeter-client-tls\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.486756 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.486727    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lbpz\" (UniqueName: \"kubernetes.io/projected/a5a629c9-0744-427c-8e88-1d4e2be4c7a9-kube-api-access-6lbpz\") pod \"telemeter-client-547fddd874-2z2wm\" (UID: \"a5a629c9-0744-427c-8e88-1d4e2be4c7a9\") " pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.487098 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.487012    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a730856-a0dc-4cfe-895f-a973ac07f2db","Type":"ContainerStarted","Data":"9b57ae2ca46c1f9ecc87c4600f6982155dc21ee77b3fafa6901445bfbd9eb155"}
Sep 06 19:17:07.487098 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.487088    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a730856-a0dc-4cfe-895f-a973ac07f2db","Type":"ContainerStarted","Data":"4229f90d838301aa19b679a8f0adc38c9f293dd6c8c2f49ac674c812f30bdb45"}
Sep 06 19:17:07.487203 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.487135    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a730856-a0dc-4cfe-895f-a973ac07f2db","Type":"ContainerStarted","Data":"77223f8f4567b8c2e8d9dd8473d92ec4bb2584d9b3e8f78170e377333f864b0a"}
Sep 06 19:17:07.487203 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.487175    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a730856-a0dc-4cfe-895f-a973ac07f2db","Type":"ContainerStarted","Data":"fb32949e737e4698260d8120e99aa1e76618dbc307a528060d28fec2afcca514"}
Sep 06 19:17:07.487293 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.487216    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a730856-a0dc-4cfe-895f-a973ac07f2db","Type":"ContainerStarted","Data":"4b1c5d2d57e5acec29f9aadf687c0b8d62bd4491fa5fdad433e467287abd6dc2"}
Sep 06 19:17:07.487293 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.487257    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a730856-a0dc-4cfe-895f-a973ac07f2db","Type":"ContainerStarted","Data":"af981f54308ce6471c68dfe16d6e397c5cbdf32325a73a23f1137264866eb12c"}
Sep 06 19:17:07.516770 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.515397    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.496542132 podStartE2EDuration="2.515379145s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.837013ms" isStatefulPod=false podCreationTimestamp="2026-09-06 19:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:17:07.514514207 +0000 UTC m=+361.695696015" watchObservedRunningTime="2026-09-06 19:17:07.515379145 +0000 UTC m=+361.696560952"
Sep 06 19:17:07.655650 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.655551    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm"
Sep 06 19:17:07.948567 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.948519    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-547fddd874-2z2wm"]
Sep 06 19:17:07.951259 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:17:07.950347    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a629c9_0744_427c_8e88_1d4e2be4c7a9.slice/crio-1c9b3c62d3bb0ceec86bfe1ca16b5283e11477ab7881fea8ceacbc8002d80ed9 WatchSource:0}: Error finding container 1c9b3c62d3bb0ceec86bfe1ca16b5283e11477ab7881fea8ceacbc8002d80ed9: Status 404 returned error can't find the container with id 1c9b3c62d3bb0ceec86bfe1ca16b5283e11477ab7881fea8ceacbc8002d80ed9
Sep 06 19:17:07.956046 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:07.956025    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 19:17:08.493864 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:08.493797    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm" event={"ID":"a5a629c9-0744-427c-8e88-1d4e2be4c7a9","Type":"ContainerStarted","Data":"1c9b3c62d3bb0ceec86bfe1ca16b5283e11477ab7881fea8ceacbc8002d80ed9"}
Sep 06 19:17:10.504935 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.504901    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm" event={"ID":"a5a629c9-0744-427c-8e88-1d4e2be4c7a9","Type":"ContainerStarted","Data":"7f11ffa57289a30a3b338ee25c3dc97fb538def3be044f8a9e819073e43961c8"}
Sep 06 19:17:10.505328 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.504954    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm" event={"ID":"a5a629c9-0744-427c-8e88-1d4e2be4c7a9","Type":"ContainerStarted","Data":"30ef0164d56ff3c51d4a7a5c27eb3edeb6ce781be679eda9de0573b505a7e922"}
Sep 06 19:17:10.505328 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.504973    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm" event={"ID":"a5a629c9-0744-427c-8e88-1d4e2be4c7a9","Type":"ContainerStarted","Data":"423d33b34f42c01fb4c960ab4941b40381954c8effa3ed752b5419f0255c2cd4"}
Sep 06 19:17:10.526741 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.526696    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-547fddd874-2z2wm" podStartSLOduration=1.9164889889999999 podStartE2EDuration="3.526679918s" totalImagesPullingTime="1.610190929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:17:10.52413448 +0000 UTC m=+364.705316296" watchObservedRunningTime="2026-09-06 19:17:10.526679918 +0000 UTC m=+364.707861737"
Sep 06 19:17:10.596027 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.595997    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5647c59464-6z8s5"]
Sep 06 19:17:10.599199 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.599180    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:10.609895 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.609872    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5647c59464-6z8s5"]
Sep 06 19:17:10.711650 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.711625    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhfzj\" (UniqueName: \"kubernetes.io/projected/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-kube-api-access-dhfzj\") pod \"console-5647c59464-6z8s5\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") " pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:10.711761 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.711658    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-console-oauth-config\") pod \"console-5647c59464-6z8s5\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") " pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:10.711761 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.711691    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-oauth-serving-cert\") pod \"console-5647c59464-6z8s5\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") " pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:10.711761 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.711740    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-service-ca\") pod \"console-5647c59464-6z8s5\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") " pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:10.711860 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.711773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-console-serving-cert\") pod \"console-5647c59464-6z8s5\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") " pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:10.711860 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.711804    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-console-config\") pod \"console-5647c59464-6z8s5\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") " pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:10.711860 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.711832    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-trusted-ca-bundle\") pod \"console-5647c59464-6z8s5\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") " pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:10.812171 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.812107    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-console-serving-cert\") pod \"console-5647c59464-6z8s5\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") " pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:10.812171 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.812137    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-console-config\") pod \"console-5647c59464-6z8s5\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") " pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:10.812171 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.812155    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-trusted-ca-bundle\") pod \"console-5647c59464-6z8s5\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") " pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:10.812338 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.812189    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhfzj\" (UniqueName: \"kubernetes.io/projected/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-kube-api-access-dhfzj\") pod \"console-5647c59464-6z8s5\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") " pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:10.812338 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.812212    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-console-oauth-config\") pod \"console-5647c59464-6z8s5\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") " pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:10.812338 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.812247    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-oauth-serving-cert\") pod \"console-5647c59464-6z8s5\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") " pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:10.812338 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.812267    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-service-ca\") pod \"console-5647c59464-6z8s5\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") " pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:10.812983 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.812960    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-service-ca\") pod \"console-5647c59464-6z8s5\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") " pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:10.813074 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.813026    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-trusted-ca-bundle\") pod \"console-5647c59464-6z8s5\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") " pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:10.813225 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.813204    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-console-config\") pod \"console-5647c59464-6z8s5\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") " pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:10.813272 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.813249    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-oauth-serving-cert\") pod \"console-5647c59464-6z8s5\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") " pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:10.814859 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.814834    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-console-oauth-config\") pod \"console-5647c59464-6z8s5\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") " pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:10.814859 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.814853    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-console-serving-cert\") pod \"console-5647c59464-6z8s5\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") " pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:10.819916 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.819899    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhfzj\" (UniqueName: \"kubernetes.io/projected/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-kube-api-access-dhfzj\") pod \"console-5647c59464-6z8s5\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") " pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:10.907927 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:10.907902    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:11.178315 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:11.178239    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5647c59464-6z8s5"]
Sep 06 19:17:11.184118 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:17:11.184094    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e86fba_59c9_400e_8c25_47cdb2ce8e13.slice/crio-b6343609116ba5659038c8d4068ca8e3dfc3a329488a18e2dbcee05e6d52245c WatchSource:0}: Error finding container b6343609116ba5659038c8d4068ca8e3dfc3a329488a18e2dbcee05e6d52245c: Status 404 returned error can't find the container with id b6343609116ba5659038c8d4068ca8e3dfc3a329488a18e2dbcee05e6d52245c
Sep 06 19:17:11.518432 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:11.518390    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5647c59464-6z8s5" event={"ID":"d1e86fba-59c9-400e-8c25-47cdb2ce8e13","Type":"ContainerStarted","Data":"0c4c0eeb051f07d84be03b69daf818ccd8b7821fa595e7a9555c602d59352e7b"}
Sep 06 19:17:11.518912 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:11.518493    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5647c59464-6z8s5" event={"ID":"d1e86fba-59c9-400e-8c25-47cdb2ce8e13","Type":"ContainerStarted","Data":"b6343609116ba5659038c8d4068ca8e3dfc3a329488a18e2dbcee05e6d52245c"}
Sep 06 19:17:20.188726 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:20.188691    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs\") pod \"network-metrics-daemon-qlvsw\" (UID: \"83a7379b-ab58-4439-998e-448e48150f3b\") " pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:17:20.191056 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:20.191028    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a7379b-ab58-4439-998e-448e48150f3b-metrics-certs\") pod \"network-metrics-daemon-qlvsw\" (UID: \"83a7379b-ab58-4439-998e-448e48150f3b\") " pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:17:20.407281 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:20.407258    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6p7v7\""
Sep 06 19:17:20.415308 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:20.415292    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlvsw"
Sep 06 19:17:20.683368 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:20.683317    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5647c59464-6z8s5" podStartSLOduration=10.683300634 podStartE2EDuration="10.683300634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:17:11.536072568 +0000 UTC m=+365.717254408" watchObservedRunningTime="2026-09-06 19:17:20.683300634 +0000 UTC m=+374.864482444"
Sep 06 19:17:20.683524 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:20.683451    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qlvsw"]
Sep 06 19:17:20.686403 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:17:20.686378    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a7379b_ab58_4439_998e_448e48150f3b.slice/crio-99b46405d008d226bfcccabce66c9a58926bd3e36e4dfbd81f0e345d4ec38fc9 WatchSource:0}: Error finding container 99b46405d008d226bfcccabce66c9a58926bd3e36e4dfbd81f0e345d4ec38fc9: Status 404 returned error can't find the container with id 99b46405d008d226bfcccabce66c9a58926bd3e36e4dfbd81f0e345d4ec38fc9
Sep 06 19:17:20.908686 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:20.908588    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:20.908686 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:20.908674    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:20.913330 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:20.913310    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:21.556100 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:21.556056    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qlvsw" event={"ID":"83a7379b-ab58-4439-998e-448e48150f3b","Type":"ContainerStarted","Data":"99b46405d008d226bfcccabce66c9a58926bd3e36e4dfbd81f0e345d4ec38fc9"}
Sep 06 19:17:21.559942 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:21.559918    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:17:21.604378 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:21.603865    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65bcfcb85b-4qk6r"]
Sep 06 19:17:22.564116 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:22.564081    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qlvsw" event={"ID":"83a7379b-ab58-4439-998e-448e48150f3b","Type":"ContainerStarted","Data":"1ac90a0b5bfb9816e4ac01f4e446a26b6b6505b8ce5022960d7b47cff97c79f2"}
Sep 06 19:17:22.564623 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:22.564130    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qlvsw" event={"ID":"83a7379b-ab58-4439-998e-448e48150f3b","Type":"ContainerStarted","Data":"00f3b7762f1ffa524b3769d32edeec67c6dfe277459fbb3cc6fb86d08d928b63"}
Sep 06 19:17:22.579558 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:22.579512    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qlvsw" podStartSLOduration=375.687505848 podStartE2EDuration="6m16.579497586s" totalImagesPullingTime="891.991738ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:17:22.57886525 +0000 UTC m=+376.760047069" watchObservedRunningTime="2026-09-06 19:17:22.579497586 +0000 UTC m=+376.760679392"
Sep 06 19:17:46.632360 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:46.632287    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65bcfcb85b-4qk6r" podUID="b93bde3b-ed23-4b7f-b04e-9a80b1883382" containerName="console" containerID="cri-o://a1aed2f3d84e72afeabf94e5365d4dfccb562acf6a9b051981aa3d0101271957" gracePeriod=15
Sep 06 19:17:46.877165 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:46.877142    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:17:47.002471 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.002435    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-oauth-serving-cert\") pod \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") "
Sep 06 19:17:47.002682 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.002542    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-console-config\") pod \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") "
Sep 06 19:17:47.002682 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.002571    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b93bde3b-ed23-4b7f-b04e-9a80b1883382-console-oauth-config\") pod \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") "
Sep 06 19:17:47.002803 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.002697    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tv7w\" (UniqueName: \"kubernetes.io/projected/b93bde3b-ed23-4b7f-b04e-9a80b1883382-kube-api-access-8tv7w\") pod \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") "
Sep 06 19:17:47.002803 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.002753    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b93bde3b-ed23-4b7f-b04e-9a80b1883382-console-serving-cert\") pod \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") "
Sep 06 19:17:47.002803 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.002798    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-service-ca\") pod \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") "
Sep 06 19:17:47.002939 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.002796    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b93bde3b-ed23-4b7f-b04e-9a80b1883382" (UID: "b93bde3b-ed23-4b7f-b04e-9a80b1883382"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:17:47.002939 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.002832    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-trusted-ca-bundle\") pod \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\" (UID: \"b93bde3b-ed23-4b7f-b04e-9a80b1883382\") "
Sep 06 19:17:47.003047 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.002933    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-console-config" (OuterVolumeSpecName: "console-config") pod "b93bde3b-ed23-4b7f-b04e-9a80b1883382" (UID: "b93bde3b-ed23-4b7f-b04e-9a80b1883382"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:17:47.003146 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.003123    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b93bde3b-ed23-4b7f-b04e-9a80b1883382" (UID: "b93bde3b-ed23-4b7f-b04e-9a80b1883382"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:17:47.003247 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.003146    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-oauth-serving-cert\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:17:47.003247 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.003169    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-console-config\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:17:47.003247 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.003173    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-service-ca" (OuterVolumeSpecName: "service-ca") pod "b93bde3b-ed23-4b7f-b04e-9a80b1883382" (UID: "b93bde3b-ed23-4b7f-b04e-9a80b1883382"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:17:47.005075 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.005052    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93bde3b-ed23-4b7f-b04e-9a80b1883382-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b93bde3b-ed23-4b7f-b04e-9a80b1883382" (UID: "b93bde3b-ed23-4b7f-b04e-9a80b1883382"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:17:47.005161 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.005085    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93bde3b-ed23-4b7f-b04e-9a80b1883382-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b93bde3b-ed23-4b7f-b04e-9a80b1883382" (UID: "b93bde3b-ed23-4b7f-b04e-9a80b1883382"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:17:47.005161 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.005118    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93bde3b-ed23-4b7f-b04e-9a80b1883382-kube-api-access-8tv7w" (OuterVolumeSpecName: "kube-api-access-8tv7w") pod "b93bde3b-ed23-4b7f-b04e-9a80b1883382" (UID: "b93bde3b-ed23-4b7f-b04e-9a80b1883382"). InnerVolumeSpecName "kube-api-access-8tv7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:17:47.104332 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.104304    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b93bde3b-ed23-4b7f-b04e-9a80b1883382-console-oauth-config\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:17:47.104332 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.104327    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tv7w\" (UniqueName: \"kubernetes.io/projected/b93bde3b-ed23-4b7f-b04e-9a80b1883382-kube-api-access-8tv7w\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:17:47.104332 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.104336    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b93bde3b-ed23-4b7f-b04e-9a80b1883382-console-serving-cert\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:17:47.104525 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.104345    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-service-ca\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:17:47.104525 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.104355    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93bde3b-ed23-4b7f-b04e-9a80b1883382-trusted-ca-bundle\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:17:47.657436 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.657401    2566 generic.go:356] "Generic (PLEG): container finished" podID="b93bde3b-ed23-4b7f-b04e-9a80b1883382" containerID="a1aed2f3d84e72afeabf94e5365d4dfccb562acf6a9b051981aa3d0101271957" exitCode=0
Sep 06 19:17:47.657854 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.657441    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65bcfcb85b-4qk6r" event={"ID":"b93bde3b-ed23-4b7f-b04e-9a80b1883382","Type":"ContainerDied","Data":"a1aed2f3d84e72afeabf94e5365d4dfccb562acf6a9b051981aa3d0101271957"}
Sep 06 19:17:47.657854 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.657457    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65bcfcb85b-4qk6r"
Sep 06 19:17:47.657854 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.657490    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65bcfcb85b-4qk6r" event={"ID":"b93bde3b-ed23-4b7f-b04e-9a80b1883382","Type":"ContainerDied","Data":"f81552dc1b3155d7bdb2b4d5c25fbae12bff2f6c033b29d95dab520ffc7c692e"}
Sep 06 19:17:47.657854 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.657511    2566 scope.go:117] "RemoveContainer" containerID="a1aed2f3d84e72afeabf94e5365d4dfccb562acf6a9b051981aa3d0101271957"
Sep 06 19:17:47.665846 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.665829    2566 scope.go:117] "RemoveContainer" containerID="a1aed2f3d84e72afeabf94e5365d4dfccb562acf6a9b051981aa3d0101271957"
Sep 06 19:17:47.666105 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:17:47.666085    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1aed2f3d84e72afeabf94e5365d4dfccb562acf6a9b051981aa3d0101271957\": container with ID starting with a1aed2f3d84e72afeabf94e5365d4dfccb562acf6a9b051981aa3d0101271957 not found: ID does not exist" containerID="a1aed2f3d84e72afeabf94e5365d4dfccb562acf6a9b051981aa3d0101271957"
Sep 06 19:17:47.666141 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.666114    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1aed2f3d84e72afeabf94e5365d4dfccb562acf6a9b051981aa3d0101271957"} err="failed to get container status \"a1aed2f3d84e72afeabf94e5365d4dfccb562acf6a9b051981aa3d0101271957\": rpc error: code = NotFound desc = could not find container \"a1aed2f3d84e72afeabf94e5365d4dfccb562acf6a9b051981aa3d0101271957\": container with ID starting with a1aed2f3d84e72afeabf94e5365d4dfccb562acf6a9b051981aa3d0101271957 not found: ID does not exist"
Sep 06 19:17:47.677040 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.677017    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65bcfcb85b-4qk6r"]
Sep 06 19:17:47.678864 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:47.678846    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65bcfcb85b-4qk6r"]
Sep 06 19:17:48.309173 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:17:48.309141    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b93bde3b-ed23-4b7f-b04e-9a80b1883382" path="/var/lib/kubelet/pods/b93bde3b-ed23-4b7f-b04e-9a80b1883382/volumes"
Sep 06 19:18:26.088224 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.088187    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f65f49f84-d62pt"]
Sep 06 19:18:26.088643 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.088492    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b93bde3b-ed23-4b7f-b04e-9a80b1883382" containerName="console"
Sep 06 19:18:26.088643 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.088502    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b93bde3b-ed23-4b7f-b04e-9a80b1883382" containerName="console"
Sep 06 19:18:26.088643 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.088561    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b93bde3b-ed23-4b7f-b04e-9a80b1883382" containerName="console"
Sep 06 19:18:26.092848 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.092832    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:26.104855 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.103030    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f65f49f84-d62pt"]
Sep 06 19:18:26.209184 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.209145    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04714b33-d678-4645-bc98-409546ff354a-console-serving-cert\") pod \"console-6f65f49f84-d62pt\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") " pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:26.209184 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.209184    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwbfw\" (UniqueName: \"kubernetes.io/projected/04714b33-d678-4645-bc98-409546ff354a-kube-api-access-zwbfw\") pod \"console-6f65f49f84-d62pt\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") " pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:26.209381 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.209208    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-oauth-serving-cert\") pod \"console-6f65f49f84-d62pt\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") " pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:26.209381 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.209256    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-trusted-ca-bundle\") pod \"console-6f65f49f84-d62pt\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") " pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:26.209381 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.209316    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04714b33-d678-4645-bc98-409546ff354a-console-oauth-config\") pod \"console-6f65f49f84-d62pt\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") " pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:26.209381 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.209333    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-console-config\") pod \"console-6f65f49f84-d62pt\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") " pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:26.209381 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.209350    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-service-ca\") pod \"console-6f65f49f84-d62pt\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") " pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:26.310370 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.310338    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04714b33-d678-4645-bc98-409546ff354a-console-serving-cert\") pod \"console-6f65f49f84-d62pt\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") " pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:26.310370 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.310370    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwbfw\" (UniqueName: \"kubernetes.io/projected/04714b33-d678-4645-bc98-409546ff354a-kube-api-access-zwbfw\") pod \"console-6f65f49f84-d62pt\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") " pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:26.310620 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.310394    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-oauth-serving-cert\") pod \"console-6f65f49f84-d62pt\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") " pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:26.310620 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.310421    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-trusted-ca-bundle\") pod \"console-6f65f49f84-d62pt\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") " pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:26.310620 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.310454    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04714b33-d678-4645-bc98-409546ff354a-console-oauth-config\") pod \"console-6f65f49f84-d62pt\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") " pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:26.310620 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.310476    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-console-config\") pod \"console-6f65f49f84-d62pt\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") " pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:26.310620 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.310503    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-service-ca\") pod \"console-6f65f49f84-d62pt\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") " pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:26.311236 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.311210    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-console-config\") pod \"console-6f65f49f84-d62pt\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") " pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:26.311236 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.311227    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-service-ca\") pod \"console-6f65f49f84-d62pt\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") " pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:26.311374 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.311227    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-oauth-serving-cert\") pod \"console-6f65f49f84-d62pt\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") " pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:26.311635 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.311591    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-trusted-ca-bundle\") pod \"console-6f65f49f84-d62pt\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") " pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:26.312920 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.312892    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04714b33-d678-4645-bc98-409546ff354a-console-serving-cert\") pod \"console-6f65f49f84-d62pt\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") " pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:26.313008 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.312965    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04714b33-d678-4645-bc98-409546ff354a-console-oauth-config\") pod \"console-6f65f49f84-d62pt\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") " pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:26.317408 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.317392    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwbfw\" (UniqueName: \"kubernetes.io/projected/04714b33-d678-4645-bc98-409546ff354a-kube-api-access-zwbfw\") pod \"console-6f65f49f84-d62pt\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") " pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:26.407545 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.407497    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:26.676178 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.676153    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f65f49f84-d62pt"]
Sep 06 19:18:26.678420 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:18:26.678399    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04714b33_d678_4645_bc98_409546ff354a.slice/crio-14a628c3999e1a970de6b040aa28dc4091674e5a1baf3a6e52ceda79f0335436 WatchSource:0}: Error finding container 14a628c3999e1a970de6b040aa28dc4091674e5a1baf3a6e52ceda79f0335436: Status 404 returned error can't find the container with id 14a628c3999e1a970de6b040aa28dc4091674e5a1baf3a6e52ceda79f0335436
Sep 06 19:18:26.804451 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.804413    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f65f49f84-d62pt" event={"ID":"04714b33-d678-4645-bc98-409546ff354a","Type":"ContainerStarted","Data":"8e0ba26bcdb74cd6089a20b43934b7f74b3fdbf768938cd239ab083c99e0731d"}
Sep 06 19:18:26.804586 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.804462    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f65f49f84-d62pt" event={"ID":"04714b33-d678-4645-bc98-409546ff354a","Type":"ContainerStarted","Data":"14a628c3999e1a970de6b040aa28dc4091674e5a1baf3a6e52ceda79f0335436"}
Sep 06 19:18:26.820837 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:26.820792    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f65f49f84-d62pt" podStartSLOduration=0.820776528 podStartE2EDuration="820.776528ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:18:26.81967301 +0000 UTC m=+441.000854816" watchObservedRunningTime="2026-09-06 19:18:26.820776528 +0000 UTC m=+441.001958334"
Sep 06 19:18:36.408441 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:36.408362    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:36.408441 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:36.408410    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:36.412793 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:36.412771    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:36.849105 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:36.849077    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:18:36.895586 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:18:36.895558    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5647c59464-6z8s5"]
Sep 06 19:19:01.917538 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:01.917502    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5647c59464-6z8s5" podUID="d1e86fba-59c9-400e-8c25-47cdb2ce8e13" containerName="console" containerID="cri-o://0c4c0eeb051f07d84be03b69daf818ccd8b7821fa595e7a9555c602d59352e7b" gracePeriod=15
Sep 06 19:19:02.159272 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.159250    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:19:02.179561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.179343    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhfzj\" (UniqueName: \"kubernetes.io/projected/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-kube-api-access-dhfzj\") pod \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") "
Sep 06 19:19:02.179561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.179388    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-oauth-serving-cert\") pod \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") "
Sep 06 19:19:02.179561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.179420    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-console-serving-cert\") pod \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") "
Sep 06 19:19:02.179561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.179442    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-trusted-ca-bundle\") pod \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") "
Sep 06 19:19:02.179561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.179479    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-service-ca\") pod \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") "
Sep 06 19:19:02.179561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.179505    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-console-config\") pod \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") "
Sep 06 19:19:02.179975 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.179596    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-console-oauth-config\") pod \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\" (UID: \"d1e86fba-59c9-400e-8c25-47cdb2ce8e13\") "
Sep 06 19:19:02.180078 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.180052    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d1e86fba-59c9-400e-8c25-47cdb2ce8e13" (UID: "d1e86fba-59c9-400e-8c25-47cdb2ce8e13"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:19:02.180139 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.180108    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-console-config" (OuterVolumeSpecName: "console-config") pod "d1e86fba-59c9-400e-8c25-47cdb2ce8e13" (UID: "d1e86fba-59c9-400e-8c25-47cdb2ce8e13"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:19:02.180139 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.180115    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d1e86fba-59c9-400e-8c25-47cdb2ce8e13" (UID: "d1e86fba-59c9-400e-8c25-47cdb2ce8e13"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:19:02.180238 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.180203    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-service-ca" (OuterVolumeSpecName: "service-ca") pod "d1e86fba-59c9-400e-8c25-47cdb2ce8e13" (UID: "d1e86fba-59c9-400e-8c25-47cdb2ce8e13"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:19:02.181834 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.181803    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d1e86fba-59c9-400e-8c25-47cdb2ce8e13" (UID: "d1e86fba-59c9-400e-8c25-47cdb2ce8e13"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:19:02.181932 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.181892    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-kube-api-access-dhfzj" (OuterVolumeSpecName: "kube-api-access-dhfzj") pod "d1e86fba-59c9-400e-8c25-47cdb2ce8e13" (UID: "d1e86fba-59c9-400e-8c25-47cdb2ce8e13"). InnerVolumeSpecName "kube-api-access-dhfzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:19:02.182292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.182279    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d1e86fba-59c9-400e-8c25-47cdb2ce8e13" (UID: "d1e86fba-59c9-400e-8c25-47cdb2ce8e13"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:19:02.280957 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.280924    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-console-oauth-config\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:19:02.280957 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.280951    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhfzj\" (UniqueName: \"kubernetes.io/projected/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-kube-api-access-dhfzj\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:19:02.281119 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.280965    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-oauth-serving-cert\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:19:02.281119 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.280979    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-console-serving-cert\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:19:02.281119 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.280991    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-trusted-ca-bundle\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:19:02.281119 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.281002    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-service-ca\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:19:02.281119 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.281016    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1e86fba-59c9-400e-8c25-47cdb2ce8e13-console-config\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:19:02.950614 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.950567    2566 generic.go:356] "Generic (PLEG): container finished" podID="d1e86fba-59c9-400e-8c25-47cdb2ce8e13" containerID="0c4c0eeb051f07d84be03b69daf818ccd8b7821fa595e7a9555c602d59352e7b" exitCode=0
Sep 06 19:19:02.951068 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.950628    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5647c59464-6z8s5" event={"ID":"d1e86fba-59c9-400e-8c25-47cdb2ce8e13","Type":"ContainerDied","Data":"0c4c0eeb051f07d84be03b69daf818ccd8b7821fa595e7a9555c602d59352e7b"}
Sep 06 19:19:02.951068 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.950661    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5647c59464-6z8s5"
Sep 06 19:19:02.951068 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.950678    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5647c59464-6z8s5" event={"ID":"d1e86fba-59c9-400e-8c25-47cdb2ce8e13","Type":"ContainerDied","Data":"b6343609116ba5659038c8d4068ca8e3dfc3a329488a18e2dbcee05e6d52245c"}
Sep 06 19:19:02.951068 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.950704    2566 scope.go:117] "RemoveContainer" containerID="0c4c0eeb051f07d84be03b69daf818ccd8b7821fa595e7a9555c602d59352e7b"
Sep 06 19:19:02.958853 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.958837    2566 scope.go:117] "RemoveContainer" containerID="0c4c0eeb051f07d84be03b69daf818ccd8b7821fa595e7a9555c602d59352e7b"
Sep 06 19:19:02.959151 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:19:02.959134    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c4c0eeb051f07d84be03b69daf818ccd8b7821fa595e7a9555c602d59352e7b\": container with ID starting with 0c4c0eeb051f07d84be03b69daf818ccd8b7821fa595e7a9555c602d59352e7b not found: ID does not exist" containerID="0c4c0eeb051f07d84be03b69daf818ccd8b7821fa595e7a9555c602d59352e7b"
Sep 06 19:19:02.959198 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.959164    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4c0eeb051f07d84be03b69daf818ccd8b7821fa595e7a9555c602d59352e7b"} err="failed to get container status \"0c4c0eeb051f07d84be03b69daf818ccd8b7821fa595e7a9555c602d59352e7b\": rpc error: code = NotFound desc = could not find container \"0c4c0eeb051f07d84be03b69daf818ccd8b7821fa595e7a9555c602d59352e7b\": container with ID starting with 0c4c0eeb051f07d84be03b69daf818ccd8b7821fa595e7a9555c602d59352e7b not found: ID does not exist"
Sep 06 19:19:02.966125 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.966103    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5647c59464-6z8s5"]
Sep 06 19:19:02.969166 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:02.969147    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5647c59464-6z8s5"]
Sep 06 19:19:04.313501 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:04.313463    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1e86fba-59c9-400e-8c25-47cdb2ce8e13" path="/var/lib/kubelet/pods/d1e86fba-59c9-400e-8c25-47cdb2ce8e13/volumes"
Sep 06 19:19:29.976024 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:29.975988    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq"]
Sep 06 19:19:29.976442 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:29.976301    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e86fba-59c9-400e-8c25-47cdb2ce8e13" containerName="console"
Sep 06 19:19:29.976442 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:29.976313    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e86fba-59c9-400e-8c25-47cdb2ce8e13" containerName="console"
Sep 06 19:19:29.976442 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:29.976367    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1e86fba-59c9-400e-8c25-47cdb2ce8e13" containerName="console"
Sep 06 19:19:29.979350 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:29.979335    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq"
Sep 06 19:19:29.982048 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:29.982026    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 06 19:19:29.982048 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:29.982039    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 06 19:19:29.982223 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:29.982171    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wrlv6\""
Sep 06 19:19:29.989701 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:29.989679    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq"]
Sep 06 19:19:30.081054 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:30.081020    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v28vv\" (UniqueName: \"kubernetes.io/projected/aa153f1b-a16c-4025-9735-8d3872d2e661-kube-api-access-v28vv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq\" (UID: \"aa153f1b-a16c-4025-9735-8d3872d2e661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq"
Sep 06 19:19:30.081238 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:30.081067    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa153f1b-a16c-4025-9735-8d3872d2e661-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq\" (UID: \"aa153f1b-a16c-4025-9735-8d3872d2e661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq"
Sep 06 19:19:30.081238 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:30.081134    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa153f1b-a16c-4025-9735-8d3872d2e661-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq\" (UID: \"aa153f1b-a16c-4025-9735-8d3872d2e661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq"
Sep 06 19:19:30.182083 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:30.182039    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa153f1b-a16c-4025-9735-8d3872d2e661-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq\" (UID: \"aa153f1b-a16c-4025-9735-8d3872d2e661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq"
Sep 06 19:19:30.182083 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:30.182093    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa153f1b-a16c-4025-9735-8d3872d2e661-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq\" (UID: \"aa153f1b-a16c-4025-9735-8d3872d2e661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq"
Sep 06 19:19:30.182312 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:30.182148    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v28vv\" (UniqueName: \"kubernetes.io/projected/aa153f1b-a16c-4025-9735-8d3872d2e661-kube-api-access-v28vv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq\" (UID: \"aa153f1b-a16c-4025-9735-8d3872d2e661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq"
Sep 06 19:19:30.182440 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:30.182416    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa153f1b-a16c-4025-9735-8d3872d2e661-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq\" (UID: \"aa153f1b-a16c-4025-9735-8d3872d2e661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq"
Sep 06 19:19:30.182495 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:30.182469    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa153f1b-a16c-4025-9735-8d3872d2e661-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq\" (UID: \"aa153f1b-a16c-4025-9735-8d3872d2e661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq"
Sep 06 19:19:30.190367 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:30.190343    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v28vv\" (UniqueName: \"kubernetes.io/projected/aa153f1b-a16c-4025-9735-8d3872d2e661-kube-api-access-v28vv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq\" (UID: \"aa153f1b-a16c-4025-9735-8d3872d2e661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq"
Sep 06 19:19:30.288596 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:30.288536    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq"
Sep 06 19:19:30.560433 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:30.560362    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq"]
Sep 06 19:19:30.563128 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:19:30.563099    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa153f1b_a16c_4025_9735_8d3872d2e661.slice/crio-e63f5016e32059a87eeafb9da71759d107e24fa77b745ed1ae6a1ca8ce6e59fa WatchSource:0}: Error finding container e63f5016e32059a87eeafb9da71759d107e24fa77b745ed1ae6a1ca8ce6e59fa: Status 404 returned error can't find the container with id e63f5016e32059a87eeafb9da71759d107e24fa77b745ed1ae6a1ca8ce6e59fa
Sep 06 19:19:31.061994 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:31.061959    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq" event={"ID":"aa153f1b-a16c-4025-9735-8d3872d2e661","Type":"ContainerStarted","Data":"e63f5016e32059a87eeafb9da71759d107e24fa77b745ed1ae6a1ca8ce6e59fa"}
Sep 06 19:19:37.087789 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:37.087753    2566 generic.go:356] "Generic (PLEG): container finished" podID="aa153f1b-a16c-4025-9735-8d3872d2e661" containerID="b53104f7e3200e4f58b3b4162abce84e98511805a03746f07b8c1003b72939f4" exitCode=0
Sep 06 19:19:37.088142 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:37.087808    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq" event={"ID":"aa153f1b-a16c-4025-9735-8d3872d2e661","Type":"ContainerDied","Data":"b53104f7e3200e4f58b3b4162abce84e98511805a03746f07b8c1003b72939f4"}
Sep 06 19:19:40.102386 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:40.102351    2566 generic.go:356] "Generic (PLEG): container finished" podID="aa153f1b-a16c-4025-9735-8d3872d2e661" containerID="ce619b523a4eba9669e390746782cc290a5df2eb33ea57af1e65e594572c33fa" exitCode=0
Sep 06 19:19:40.102793 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:40.102436    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq" event={"ID":"aa153f1b-a16c-4025-9735-8d3872d2e661","Type":"ContainerDied","Data":"ce619b523a4eba9669e390746782cc290a5df2eb33ea57af1e65e594572c33fa"}
Sep 06 19:19:47.138004 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:47.137925    2566 generic.go:356] "Generic (PLEG): container finished" podID="aa153f1b-a16c-4025-9735-8d3872d2e661" containerID="3e877c99638592095fc82d07b774fe2bc9c83f336b7ede2f48b7636eda9dae64" exitCode=0
Sep 06 19:19:47.138004 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:47.137971    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq" event={"ID":"aa153f1b-a16c-4025-9735-8d3872d2e661","Type":"ContainerDied","Data":"3e877c99638592095fc82d07b774fe2bc9c83f336b7ede2f48b7636eda9dae64"}
Sep 06 19:19:48.264394 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:48.264371    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq"
Sep 06 19:19:48.344167 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:48.344135    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa153f1b-a16c-4025-9735-8d3872d2e661-util\") pod \"aa153f1b-a16c-4025-9735-8d3872d2e661\" (UID: \"aa153f1b-a16c-4025-9735-8d3872d2e661\") "
Sep 06 19:19:48.344322 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:48.344220    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa153f1b-a16c-4025-9735-8d3872d2e661-bundle\") pod \"aa153f1b-a16c-4025-9735-8d3872d2e661\" (UID: \"aa153f1b-a16c-4025-9735-8d3872d2e661\") "
Sep 06 19:19:48.344322 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:48.344255    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v28vv\" (UniqueName: \"kubernetes.io/projected/aa153f1b-a16c-4025-9735-8d3872d2e661-kube-api-access-v28vv\") pod \"aa153f1b-a16c-4025-9735-8d3872d2e661\" (UID: \"aa153f1b-a16c-4025-9735-8d3872d2e661\") "
Sep 06 19:19:48.345039 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:48.344956    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa153f1b-a16c-4025-9735-8d3872d2e661-bundle" (OuterVolumeSpecName: "bundle") pod "aa153f1b-a16c-4025-9735-8d3872d2e661" (UID: "aa153f1b-a16c-4025-9735-8d3872d2e661"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:19:48.346511 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:48.346478    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa153f1b-a16c-4025-9735-8d3872d2e661-kube-api-access-v28vv" (OuterVolumeSpecName: "kube-api-access-v28vv") pod "aa153f1b-a16c-4025-9735-8d3872d2e661" (UID: "aa153f1b-a16c-4025-9735-8d3872d2e661"). InnerVolumeSpecName "kube-api-access-v28vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:19:48.348598 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:48.348557    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa153f1b-a16c-4025-9735-8d3872d2e661-util" (OuterVolumeSpecName: "util") pod "aa153f1b-a16c-4025-9735-8d3872d2e661" (UID: "aa153f1b-a16c-4025-9735-8d3872d2e661"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:19:48.444798 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:48.444751    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa153f1b-a16c-4025-9735-8d3872d2e661-util\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:19:48.444798 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:48.444772    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa153f1b-a16c-4025-9735-8d3872d2e661-bundle\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:19:48.444798 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:48.444780    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v28vv\" (UniqueName: \"kubernetes.io/projected/aa153f1b-a16c-4025-9735-8d3872d2e661-kube-api-access-v28vv\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:19:49.148270 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:49.148235    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq" event={"ID":"aa153f1b-a16c-4025-9735-8d3872d2e661","Type":"ContainerDied","Data":"e63f5016e32059a87eeafb9da71759d107e24fa77b745ed1ae6a1ca8ce6e59fa"}
Sep 06 19:19:49.148270 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:49.148262    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407bgpq"
Sep 06 19:19:49.148460 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:49.148305    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e63f5016e32059a87eeafb9da71759d107e24fa77b745ed1ae6a1ca8ce6e59fa"
Sep 06 19:19:52.680481 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:52.680452    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tpbvs"]
Sep 06 19:19:52.680930 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:52.680770    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa153f1b-a16c-4025-9735-8d3872d2e661" containerName="util"
Sep 06 19:19:52.680930 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:52.680782    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa153f1b-a16c-4025-9735-8d3872d2e661" containerName="util"
Sep 06 19:19:52.680930 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:52.680804    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa153f1b-a16c-4025-9735-8d3872d2e661" containerName="extract"
Sep 06 19:19:52.680930 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:52.680810    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa153f1b-a16c-4025-9735-8d3872d2e661" containerName="extract"
Sep 06 19:19:52.680930 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:52.680821    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa153f1b-a16c-4025-9735-8d3872d2e661" containerName="pull"
Sep 06 19:19:52.680930 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:52.680830    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa153f1b-a16c-4025-9735-8d3872d2e661" containerName="pull"
Sep 06 19:19:52.680930 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:52.680900    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa153f1b-a16c-4025-9735-8d3872d2e661" containerName="extract"
Sep 06 19:19:52.685656 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:52.685639    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tpbvs"
Sep 06 19:19:52.688429 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:52.688409    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 06 19:19:52.688429 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:52.688417    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 06 19:19:52.688564 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:52.688470    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-klh2z\""
Sep 06 19:19:52.696362 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:52.696342    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tpbvs"]
Sep 06 19:19:52.778232 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:52.778199    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69552\" (UniqueName: \"kubernetes.io/projected/088fc6d0-eee4-401e-8051-b5a1188c01a1-kube-api-access-69552\") pod \"cert-manager-operator-controller-manager-55c444bf86-tpbvs\" (UID: \"088fc6d0-eee4-401e-8051-b5a1188c01a1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tpbvs"
Sep 06 19:19:52.778388 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:52.778246    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/088fc6d0-eee4-401e-8051-b5a1188c01a1-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-tpbvs\" (UID: \"088fc6d0-eee4-401e-8051-b5a1188c01a1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tpbvs"
Sep 06 19:19:52.879107 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:52.879075    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/088fc6d0-eee4-401e-8051-b5a1188c01a1-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-tpbvs\" (UID: \"088fc6d0-eee4-401e-8051-b5a1188c01a1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tpbvs"
Sep 06 19:19:52.879202 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:52.879143    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69552\" (UniqueName: \"kubernetes.io/projected/088fc6d0-eee4-401e-8051-b5a1188c01a1-kube-api-access-69552\") pod \"cert-manager-operator-controller-manager-55c444bf86-tpbvs\" (UID: \"088fc6d0-eee4-401e-8051-b5a1188c01a1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tpbvs"
Sep 06 19:19:52.879443 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:52.879424    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/088fc6d0-eee4-401e-8051-b5a1188c01a1-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-tpbvs\" (UID: \"088fc6d0-eee4-401e-8051-b5a1188c01a1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tpbvs"
Sep 06 19:19:52.889357 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:52.889328    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69552\" (UniqueName: \"kubernetes.io/projected/088fc6d0-eee4-401e-8051-b5a1188c01a1-kube-api-access-69552\") pod \"cert-manager-operator-controller-manager-55c444bf86-tpbvs\" (UID: \"088fc6d0-eee4-401e-8051-b5a1188c01a1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tpbvs"
Sep 06 19:19:52.995403 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:52.995378    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tpbvs"
Sep 06 19:19:53.267770 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:53.267705    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tpbvs"]
Sep 06 19:19:53.270376 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:19:53.270354    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod088fc6d0_eee4_401e_8051_b5a1188c01a1.slice/crio-b0b41065bdd40c4a3362447e212a747f461dcc20ad2b996e2d406779705a31dc WatchSource:0}: Error finding container b0b41065bdd40c4a3362447e212a747f461dcc20ad2b996e2d406779705a31dc: Status 404 returned error can't find the container with id b0b41065bdd40c4a3362447e212a747f461dcc20ad2b996e2d406779705a31dc
Sep 06 19:19:54.169481 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:54.169446    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tpbvs" event={"ID":"088fc6d0-eee4-401e-8051-b5a1188c01a1","Type":"ContainerStarted","Data":"b0b41065bdd40c4a3362447e212a747f461dcc20ad2b996e2d406779705a31dc"}
Sep 06 19:19:56.179782 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:56.179746    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tpbvs" event={"ID":"088fc6d0-eee4-401e-8051-b5a1188c01a1","Type":"ContainerStarted","Data":"2f6eb796088a80bcde6c8c520f9e1194620c4335370c78921ef32bf330ad6375"}
Sep 06 19:19:56.199242 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:56.199197    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tpbvs" podStartSLOduration=2.299055109 podStartE2EDuration="4.199181114s" totalImagesPullingTime="1.900126005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:19:56.197176247 +0000 UTC m=+530.378358053" watchObservedRunningTime="2026-09-06 19:19:56.199181114 +0000 UTC m=+530.380362920"
Sep 06 19:19:56.978949 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:56.978914    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-44h8c"]
Sep 06 19:19:56.982215 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:56.982200    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-44h8c"
Sep 06 19:19:56.984693 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:56.984671    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 06 19:19:56.985591 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:56.985571    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 06 19:19:56.985696 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:56.985571    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-hcnbc\""
Sep 06 19:19:56.991777 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:56.991747    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-44h8c"]
Sep 06 19:19:57.113405 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.113377    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70d409e1-2583-4e04-a0d3-89cf25f4b8dc-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-44h8c\" (UID: \"70d409e1-2583-4e04-a0d3-89cf25f4b8dc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-44h8c"
Sep 06 19:19:57.113558 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.113428    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj6m2\" (UniqueName: \"kubernetes.io/projected/70d409e1-2583-4e04-a0d3-89cf25f4b8dc-kube-api-access-vj6m2\") pod \"cert-manager-webhook-6486ffd997-44h8c\" (UID: \"70d409e1-2583-4e04-a0d3-89cf25f4b8dc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-44h8c"
Sep 06 19:19:57.177027 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.176993    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7"]
Sep 06 19:19:57.180348 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.180329    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7"
Sep 06 19:19:57.182990 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.182956    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 06 19:19:57.182990 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.182983    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 06 19:19:57.183139 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.183065    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wrlv6\""
Sep 06 19:19:57.188952 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.188922    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7"]
Sep 06 19:19:57.214420 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.214395    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj6m2\" (UniqueName: \"kubernetes.io/projected/70d409e1-2583-4e04-a0d3-89cf25f4b8dc-kube-api-access-vj6m2\") pod \"cert-manager-webhook-6486ffd997-44h8c\" (UID: \"70d409e1-2583-4e04-a0d3-89cf25f4b8dc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-44h8c"
Sep 06 19:19:57.214517 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.214449    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70d409e1-2583-4e04-a0d3-89cf25f4b8dc-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-44h8c\" (UID: \"70d409e1-2583-4e04-a0d3-89cf25f4b8dc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-44h8c"
Sep 06 19:19:57.222032 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.222000    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70d409e1-2583-4e04-a0d3-89cf25f4b8dc-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-44h8c\" (UID: \"70d409e1-2583-4e04-a0d3-89cf25f4b8dc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-44h8c"
Sep 06 19:19:57.222135 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.222120    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj6m2\" (UniqueName: \"kubernetes.io/projected/70d409e1-2583-4e04-a0d3-89cf25f4b8dc-kube-api-access-vj6m2\") pod \"cert-manager-webhook-6486ffd997-44h8c\" (UID: \"70d409e1-2583-4e04-a0d3-89cf25f4b8dc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-44h8c"
Sep 06 19:19:57.307479 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.307412    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-44h8c"
Sep 06 19:19:57.315648 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.315619    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b88c8d83-2974-4036-90e7-fafdb67d0f19-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7\" (UID: \"b88c8d83-2974-4036-90e7-fafdb67d0f19\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7"
Sep 06 19:19:57.315755 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.315679    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b88c8d83-2974-4036-90e7-fafdb67d0f19-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7\" (UID: \"b88c8d83-2974-4036-90e7-fafdb67d0f19\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7"
Sep 06 19:19:57.315864 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.315777    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n86hf\" (UniqueName: \"kubernetes.io/projected/b88c8d83-2974-4036-90e7-fafdb67d0f19-kube-api-access-n86hf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7\" (UID: \"b88c8d83-2974-4036-90e7-fafdb67d0f19\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7"
Sep 06 19:19:57.416225 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.416194    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b88c8d83-2974-4036-90e7-fafdb67d0f19-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7\" (UID: \"b88c8d83-2974-4036-90e7-fafdb67d0f19\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7"
Sep 06 19:19:57.416356 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.416244    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n86hf\" (UniqueName: \"kubernetes.io/projected/b88c8d83-2974-4036-90e7-fafdb67d0f19-kube-api-access-n86hf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7\" (UID: \"b88c8d83-2974-4036-90e7-fafdb67d0f19\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7"
Sep 06 19:19:57.416356 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.416291    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b88c8d83-2974-4036-90e7-fafdb67d0f19-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7\" (UID: \"b88c8d83-2974-4036-90e7-fafdb67d0f19\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7"
Sep 06 19:19:57.416588 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.416563    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b88c8d83-2974-4036-90e7-fafdb67d0f19-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7\" (UID: \"b88c8d83-2974-4036-90e7-fafdb67d0f19\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7"
Sep 06 19:19:57.416668 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.416652    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b88c8d83-2974-4036-90e7-fafdb67d0f19-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7\" (UID: \"b88c8d83-2974-4036-90e7-fafdb67d0f19\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7"
Sep 06 19:19:57.424395 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.424363    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n86hf\" (UniqueName: \"kubernetes.io/projected/b88c8d83-2974-4036-90e7-fafdb67d0f19-kube-api-access-n86hf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7\" (UID: \"b88c8d83-2974-4036-90e7-fafdb67d0f19\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7"
Sep 06 19:19:57.490464 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.490434    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7"
Sep 06 19:19:57.580016 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.579994    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-44h8c"]
Sep 06 19:19:57.582364 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:19:57.582339    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70d409e1_2583_4e04_a0d3_89cf25f4b8dc.slice/crio-781fc15199b091b6be0fe79bfb69e1446c28e58bca17682860d6094223a89ea6 WatchSource:0}: Error finding container 781fc15199b091b6be0fe79bfb69e1446c28e58bca17682860d6094223a89ea6: Status 404 returned error can't find the container with id 781fc15199b091b6be0fe79bfb69e1446c28e58bca17682860d6094223a89ea6
Sep 06 19:19:57.761019 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:57.760999    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7"]
Sep 06 19:19:57.763264 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:19:57.763237    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88c8d83_2974_4036_90e7_fafdb67d0f19.slice/crio-25aeec056e6918170af8a30a8085cdec8d78831572f980ba4485cf55457d86e6 WatchSource:0}: Error finding container 25aeec056e6918170af8a30a8085cdec8d78831572f980ba4485cf55457d86e6: Status 404 returned error can't find the container with id 25aeec056e6918170af8a30a8085cdec8d78831572f980ba4485cf55457d86e6
Sep 06 19:19:58.189431 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:58.189344    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-44h8c" event={"ID":"70d409e1-2583-4e04-a0d3-89cf25f4b8dc","Type":"ContainerStarted","Data":"781fc15199b091b6be0fe79bfb69e1446c28e58bca17682860d6094223a89ea6"}
Sep 06 19:19:58.190996 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:58.190973    2566 generic.go:356] "Generic (PLEG): container finished" podID="b88c8d83-2974-4036-90e7-fafdb67d0f19" containerID="b25b9109cbb5a58eb887435d9d9b9664a61655c261080d49df66f0d6a78fd5e9" exitCode=0
Sep 06 19:19:58.191059 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:58.191017    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7" event={"ID":"b88c8d83-2974-4036-90e7-fafdb67d0f19","Type":"ContainerDied","Data":"b25b9109cbb5a58eb887435d9d9b9664a61655c261080d49df66f0d6a78fd5e9"}
Sep 06 19:19:58.191095 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:19:58.191069    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7" event={"ID":"b88c8d83-2974-4036-90e7-fafdb67d0f19","Type":"ContainerStarted","Data":"25aeec056e6918170af8a30a8085cdec8d78831572f980ba4485cf55457d86e6"}
Sep 06 19:20:01.209799 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:01.209764    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-44h8c" event={"ID":"70d409e1-2583-4e04-a0d3-89cf25f4b8dc","Type":"ContainerStarted","Data":"32ba266402b89f7eaafe0e7e8657552281525585392e73072f62596597335d49"}
Sep 06 19:20:01.210266 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:01.210000    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-44h8c"
Sep 06 19:20:01.212114 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:01.212059    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7" event={"ID":"b88c8d83-2974-4036-90e7-fafdb67d0f19","Type":"ContainerStarted","Data":"086c1143b824076bdb5b8cf006513c63c520a799db2a4c3b107f561b554dc7a5"}
Sep 06 19:20:01.227567 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:01.227516    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-44h8c" podStartSLOduration=1.691826835 podStartE2EDuration="5.227500547s" totalImagesPullingTime="3.535673712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:20:01.225814392 +0000 UTC m=+535.406996199" watchObservedRunningTime="2026-09-06 19:20:01.227500547 +0000 UTC m=+535.408682352"
Sep 06 19:20:02.218869 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:02.218837    2566 generic.go:356] "Generic (PLEG): container finished" podID="b88c8d83-2974-4036-90e7-fafdb67d0f19" containerID="086c1143b824076bdb5b8cf006513c63c520a799db2a4c3b107f561b554dc7a5" exitCode=0
Sep 06 19:20:02.219250 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:02.218960    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7" event={"ID":"b88c8d83-2974-4036-90e7-fafdb67d0f19","Type":"ContainerDied","Data":"086c1143b824076bdb5b8cf006513c63c520a799db2a4c3b107f561b554dc7a5"}
Sep 06 19:20:03.225411 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:03.225373    2566 generic.go:356] "Generic (PLEG): container finished" podID="b88c8d83-2974-4036-90e7-fafdb67d0f19" containerID="bc0ba2cf3f1ca9a2fdc35f7304eadbde78684cc972d39a379dda2fffb1aadbba" exitCode=0
Sep 06 19:20:03.225809 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:03.225460    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7" event={"ID":"b88c8d83-2974-4036-90e7-fafdb67d0f19","Type":"ContainerDied","Data":"bc0ba2cf3f1ca9a2fdc35f7304eadbde78684cc972d39a379dda2fffb1aadbba"}
Sep 06 19:20:04.161998 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:04.161967    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-j2kll"]
Sep 06 19:20:04.165889 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:04.165870    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-j2kll"
Sep 06 19:20:04.168209 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:04.168189    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-l8k4n\""
Sep 06 19:20:04.172032 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:04.172015    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-j2kll"]
Sep 06 19:20:04.174760 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:04.174738    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/798b39fe-bf69-4c84-a6f5-c90f05639f7e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-j2kll\" (UID: \"798b39fe-bf69-4c84-a6f5-c90f05639f7e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j2kll"
Sep 06 19:20:04.174844 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:04.174782    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59dtg\" (UniqueName: \"kubernetes.io/projected/798b39fe-bf69-4c84-a6f5-c90f05639f7e-kube-api-access-59dtg\") pod \"cert-manager-cainjector-559cdd8c87-j2kll\" (UID: \"798b39fe-bf69-4c84-a6f5-c90f05639f7e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j2kll"
Sep 06 19:20:04.275487 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:04.275416    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59dtg\" (UniqueName: \"kubernetes.io/projected/798b39fe-bf69-4c84-a6f5-c90f05639f7e-kube-api-access-59dtg\") pod \"cert-manager-cainjector-559cdd8c87-j2kll\" (UID: \"798b39fe-bf69-4c84-a6f5-c90f05639f7e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j2kll"
Sep 06 19:20:04.275487 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:04.275479    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/798b39fe-bf69-4c84-a6f5-c90f05639f7e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-j2kll\" (UID: \"798b39fe-bf69-4c84-a6f5-c90f05639f7e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j2kll"
Sep 06 19:20:04.285312 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:04.285286    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/798b39fe-bf69-4c84-a6f5-c90f05639f7e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-j2kll\" (UID: \"798b39fe-bf69-4c84-a6f5-c90f05639f7e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j2kll"
Sep 06 19:20:04.285409 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:04.285386    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59dtg\" (UniqueName: \"kubernetes.io/projected/798b39fe-bf69-4c84-a6f5-c90f05639f7e-kube-api-access-59dtg\") pod \"cert-manager-cainjector-559cdd8c87-j2kll\" (UID: \"798b39fe-bf69-4c84-a6f5-c90f05639f7e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j2kll"
Sep 06 19:20:04.357702 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:04.357679    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7"
Sep 06 19:20:04.376299 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:04.376273    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b88c8d83-2974-4036-90e7-fafdb67d0f19-bundle\") pod \"b88c8d83-2974-4036-90e7-fafdb67d0f19\" (UID: \"b88c8d83-2974-4036-90e7-fafdb67d0f19\") "
Sep 06 19:20:04.376400 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:04.376354    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n86hf\" (UniqueName: \"kubernetes.io/projected/b88c8d83-2974-4036-90e7-fafdb67d0f19-kube-api-access-n86hf\") pod \"b88c8d83-2974-4036-90e7-fafdb67d0f19\" (UID: \"b88c8d83-2974-4036-90e7-fafdb67d0f19\") "
Sep 06 19:20:04.376400 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:04.376383    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b88c8d83-2974-4036-90e7-fafdb67d0f19-util\") pod \"b88c8d83-2974-4036-90e7-fafdb67d0f19\" (UID: \"b88c8d83-2974-4036-90e7-fafdb67d0f19\") "
Sep 06 19:20:04.376666 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:04.376631    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88c8d83-2974-4036-90e7-fafdb67d0f19-bundle" (OuterVolumeSpecName: "bundle") pod "b88c8d83-2974-4036-90e7-fafdb67d0f19" (UID: "b88c8d83-2974-4036-90e7-fafdb67d0f19"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:20:04.378989 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:04.378959    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88c8d83-2974-4036-90e7-fafdb67d0f19-kube-api-access-n86hf" (OuterVolumeSpecName: "kube-api-access-n86hf") pod "b88c8d83-2974-4036-90e7-fafdb67d0f19" (UID: "b88c8d83-2974-4036-90e7-fafdb67d0f19"). InnerVolumeSpecName "kube-api-access-n86hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:20:04.381072 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:04.381050    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88c8d83-2974-4036-90e7-fafdb67d0f19-util" (OuterVolumeSpecName: "util") pod "b88c8d83-2974-4036-90e7-fafdb67d0f19" (UID: "b88c8d83-2974-4036-90e7-fafdb67d0f19"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:20:04.475540 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:04.475516    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-j2kll"
Sep 06 19:20:04.476927 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:04.476910    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n86hf\" (UniqueName: \"kubernetes.io/projected/b88c8d83-2974-4036-90e7-fafdb67d0f19-kube-api-access-n86hf\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:20:04.476978 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:04.476930    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b88c8d83-2974-4036-90e7-fafdb67d0f19-util\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:20:04.476978 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:04.476940    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b88c8d83-2974-4036-90e7-fafdb67d0f19-bundle\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:20:04.742913 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:04.742891    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-j2kll"]
Sep 06 19:20:04.745213 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:20:04.745174    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod798b39fe_bf69_4c84_a6f5_c90f05639f7e.slice/crio-8139bcc94f97e358bea3c1348eb3b99c6f746580d137002d007247758d926fea WatchSource:0}: Error finding container 8139bcc94f97e358bea3c1348eb3b99c6f746580d137002d007247758d926fea: Status 404 returned error can't find the container with id 8139bcc94f97e358bea3c1348eb3b99c6f746580d137002d007247758d926fea
Sep 06 19:20:05.235234 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:05.235195    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-j2kll" event={"ID":"798b39fe-bf69-4c84-a6f5-c90f05639f7e","Type":"ContainerStarted","Data":"fcabfe6fcd48dd455d9c5b0239404ce2f3a8fdecad0a4f318e409d84e883884e"}
Sep 06 19:20:05.236198 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:05.235246    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-j2kll" event={"ID":"798b39fe-bf69-4c84-a6f5-c90f05639f7e","Type":"ContainerStarted","Data":"8139bcc94f97e358bea3c1348eb3b99c6f746580d137002d007247758d926fea"}
Sep 06 19:20:05.240480 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:05.237928    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7" event={"ID":"b88c8d83-2974-4036-90e7-fafdb67d0f19","Type":"ContainerDied","Data":"25aeec056e6918170af8a30a8085cdec8d78831572f980ba4485cf55457d86e6"}
Sep 06 19:20:05.240480 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:05.238011    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25aeec056e6918170af8a30a8085cdec8d78831572f980ba4485cf55457d86e6"
Sep 06 19:20:05.240480 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:05.238110    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp78l7"
Sep 06 19:20:05.253595 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:05.253556    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-j2kll" podStartSLOduration=1.253543114 podStartE2EDuration="1.253543114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:20:05.252324271 +0000 UTC m=+539.433506077" watchObservedRunningTime="2026-09-06 19:20:05.253543114 +0000 UTC m=+539.434724919"
Sep 06 19:20:07.221509 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:07.221474    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-44h8c"
Sep 06 19:20:16.979296 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:16.979258    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt"]
Sep 06 19:20:16.979694 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:16.979555    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b88c8d83-2974-4036-90e7-fafdb67d0f19" containerName="extract"
Sep 06 19:20:16.979694 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:16.979566    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b88c8d83-2974-4036-90e7-fafdb67d0f19" containerName="extract"
Sep 06 19:20:16.979694 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:16.979577    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b88c8d83-2974-4036-90e7-fafdb67d0f19" containerName="pull"
Sep 06 19:20:16.979694 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:16.979582    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b88c8d83-2974-4036-90e7-fafdb67d0f19" containerName="pull"
Sep 06 19:20:16.979694 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:16.979593    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b88c8d83-2974-4036-90e7-fafdb67d0f19" containerName="util"
Sep 06 19:20:16.979694 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:16.979599    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b88c8d83-2974-4036-90e7-fafdb67d0f19" containerName="util"
Sep 06 19:20:16.979694 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:16.979678    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b88c8d83-2974-4036-90e7-fafdb67d0f19" containerName="extract"
Sep 06 19:20:16.983111 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:16.983096    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt"
Sep 06 19:20:16.985450 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:16.985426    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 06 19:20:16.985580 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:16.985429    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 06 19:20:16.985580 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:16.985432    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wrlv6\""
Sep 06 19:20:16.991399 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:16.991380    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt"]
Sep 06 19:20:17.067960 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:17.067930    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4639520-9baf-4b54-a25c-96f95cf65c32-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt\" (UID: \"b4639520-9baf-4b54-a25c-96f95cf65c32\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt"
Sep 06 19:20:17.068090 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:17.067986    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4639520-9baf-4b54-a25c-96f95cf65c32-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt\" (UID: \"b4639520-9baf-4b54-a25c-96f95cf65c32\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt"
Sep 06 19:20:17.068090 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:17.068028    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl2vm\" (UniqueName: \"kubernetes.io/projected/b4639520-9baf-4b54-a25c-96f95cf65c32-kube-api-access-kl2vm\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt\" (UID: \"b4639520-9baf-4b54-a25c-96f95cf65c32\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt"
Sep 06 19:20:17.169285 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:17.169239    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4639520-9baf-4b54-a25c-96f95cf65c32-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt\" (UID: \"b4639520-9baf-4b54-a25c-96f95cf65c32\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt"
Sep 06 19:20:17.169285 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:17.169293    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl2vm\" (UniqueName: \"kubernetes.io/projected/b4639520-9baf-4b54-a25c-96f95cf65c32-kube-api-access-kl2vm\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt\" (UID: \"b4639520-9baf-4b54-a25c-96f95cf65c32\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt"
Sep 06 19:20:17.169482 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:17.169329    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4639520-9baf-4b54-a25c-96f95cf65c32-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt\" (UID: \"b4639520-9baf-4b54-a25c-96f95cf65c32\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt"
Sep 06 19:20:17.169664 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:17.169647    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4639520-9baf-4b54-a25c-96f95cf65c32-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt\" (UID: \"b4639520-9baf-4b54-a25c-96f95cf65c32\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt"
Sep 06 19:20:17.169702 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:17.169688    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4639520-9baf-4b54-a25c-96f95cf65c32-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt\" (UID: \"b4639520-9baf-4b54-a25c-96f95cf65c32\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt"
Sep 06 19:20:17.177626 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:17.177587    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl2vm\" (UniqueName: \"kubernetes.io/projected/b4639520-9baf-4b54-a25c-96f95cf65c32-kube-api-access-kl2vm\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt\" (UID: \"b4639520-9baf-4b54-a25c-96f95cf65c32\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt"
Sep 06 19:20:17.293255 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:17.293180    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt"
Sep 06 19:20:17.564172 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:17.563977    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt"]
Sep 06 19:20:17.566678 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:20:17.566645    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4639520_9baf_4b54_a25c_96f95cf65c32.slice/crio-adf839fd251bdbe424dfb0538bdd7d7bcba1176f8471fe8df6c32a00c3d71f3c WatchSource:0}: Error finding container adf839fd251bdbe424dfb0538bdd7d7bcba1176f8471fe8df6c32a00c3d71f3c: Status 404 returned error can't find the container with id adf839fd251bdbe424dfb0538bdd7d7bcba1176f8471fe8df6c32a00c3d71f3c
Sep 06 19:20:18.297576 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:18.297536    2566 generic.go:356] "Generic (PLEG): container finished" podID="b4639520-9baf-4b54-a25c-96f95cf65c32" containerID="6969cb229741d485c14e4908b73d8ff5f1c8452d867914bd2659c9c232338e62" exitCode=0
Sep 06 19:20:18.297986 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:18.297636    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt" event={"ID":"b4639520-9baf-4b54-a25c-96f95cf65c32","Type":"ContainerDied","Data":"6969cb229741d485c14e4908b73d8ff5f1c8452d867914bd2659c9c232338e62"}
Sep 06 19:20:18.297986 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:18.297684    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt" event={"ID":"b4639520-9baf-4b54-a25c-96f95cf65c32","Type":"ContainerStarted","Data":"adf839fd251bdbe424dfb0538bdd7d7bcba1176f8471fe8df6c32a00c3d71f3c"}
Sep 06 19:20:20.307879 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:20.307836    2566 generic.go:356] "Generic (PLEG): container finished" podID="b4639520-9baf-4b54-a25c-96f95cf65c32" containerID="5239ff28538aec93b27b069a7fc32ab59d20110ad895676061ef9a40bd4c7a38" exitCode=0
Sep 06 19:20:20.309309 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:20.309281    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt" event={"ID":"b4639520-9baf-4b54-a25c-96f95cf65c32","Type":"ContainerDied","Data":"5239ff28538aec93b27b069a7fc32ab59d20110ad895676061ef9a40bd4c7a38"}
Sep 06 19:20:21.315666 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:21.315598    2566 generic.go:356] "Generic (PLEG): container finished" podID="b4639520-9baf-4b54-a25c-96f95cf65c32" containerID="aaaa082490934adf21358a211b511f8a5a07d059f434f6f86b3161271477aaa9" exitCode=0
Sep 06 19:20:21.316023 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:21.315695    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt" event={"ID":"b4639520-9baf-4b54-a25c-96f95cf65c32","Type":"ContainerDied","Data":"aaaa082490934adf21358a211b511f8a5a07d059f434f6f86b3161271477aaa9"}
Sep 06 19:20:22.448052 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:22.448027    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt"
Sep 06 19:20:22.510425 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:22.510397    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kl2vm\" (UniqueName: \"kubernetes.io/projected/b4639520-9baf-4b54-a25c-96f95cf65c32-kube-api-access-kl2vm\") pod \"b4639520-9baf-4b54-a25c-96f95cf65c32\" (UID: \"b4639520-9baf-4b54-a25c-96f95cf65c32\") "
Sep 06 19:20:22.510585 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:22.510472    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4639520-9baf-4b54-a25c-96f95cf65c32-util\") pod \"b4639520-9baf-4b54-a25c-96f95cf65c32\" (UID: \"b4639520-9baf-4b54-a25c-96f95cf65c32\") "
Sep 06 19:20:22.510725 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:22.510579    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4639520-9baf-4b54-a25c-96f95cf65c32-bundle\") pod \"b4639520-9baf-4b54-a25c-96f95cf65c32\" (UID: \"b4639520-9baf-4b54-a25c-96f95cf65c32\") "
Sep 06 19:20:22.511340 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:22.511317    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4639520-9baf-4b54-a25c-96f95cf65c32-bundle" (OuterVolumeSpecName: "bundle") pod "b4639520-9baf-4b54-a25c-96f95cf65c32" (UID: "b4639520-9baf-4b54-a25c-96f95cf65c32"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:20:22.512534 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:22.512504    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4639520-9baf-4b54-a25c-96f95cf65c32-kube-api-access-kl2vm" (OuterVolumeSpecName: "kube-api-access-kl2vm") pod "b4639520-9baf-4b54-a25c-96f95cf65c32" (UID: "b4639520-9baf-4b54-a25c-96f95cf65c32"). InnerVolumeSpecName "kube-api-access-kl2vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:20:22.516188 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:22.516170    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4639520-9baf-4b54-a25c-96f95cf65c32-util" (OuterVolumeSpecName: "util") pod "b4639520-9baf-4b54-a25c-96f95cf65c32" (UID: "b4639520-9baf-4b54-a25c-96f95cf65c32"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:20:22.611809 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:22.611758    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl2vm\" (UniqueName: \"kubernetes.io/projected/b4639520-9baf-4b54-a25c-96f95cf65c32-kube-api-access-kl2vm\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:20:22.611809 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:22.611778    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4639520-9baf-4b54-a25c-96f95cf65c32-util\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:20:22.611809 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:22.611786    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4639520-9baf-4b54-a25c-96f95cf65c32-bundle\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:20:23.326989 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:23.326953    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt" event={"ID":"b4639520-9baf-4b54-a25c-96f95cf65c32","Type":"ContainerDied","Data":"adf839fd251bdbe424dfb0538bdd7d7bcba1176f8471fe8df6c32a00c3d71f3c"}
Sep 06 19:20:23.326989 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:23.326976    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6r6wt"
Sep 06 19:20:23.326989 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:23.326994    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adf839fd251bdbe424dfb0538bdd7d7bcba1176f8471fe8df6c32a00c3d71f3c"
Sep 06 19:20:33.398558 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:33.398527    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd"]
Sep 06 19:20:33.399003 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:33.398836    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4639520-9baf-4b54-a25c-96f95cf65c32" containerName="pull"
Sep 06 19:20:33.399003 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:33.398849    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4639520-9baf-4b54-a25c-96f95cf65c32" containerName="pull"
Sep 06 19:20:33.399003 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:33.398861    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4639520-9baf-4b54-a25c-96f95cf65c32" containerName="extract"
Sep 06 19:20:33.399003 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:33.398867    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4639520-9baf-4b54-a25c-96f95cf65c32" containerName="extract"
Sep 06 19:20:33.399003 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:33.398874    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4639520-9baf-4b54-a25c-96f95cf65c32" containerName="util"
Sep 06 19:20:33.399003 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:33.398880    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4639520-9baf-4b54-a25c-96f95cf65c32" containerName="util"
Sep 06 19:20:33.399003 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:33.398932    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4639520-9baf-4b54-a25c-96f95cf65c32" containerName="extract"
Sep 06 19:20:33.437911 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:33.437889    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd"]
Sep 06 19:20:33.438043 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:33.437974    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd"
Sep 06 19:20:33.440519 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:33.440497    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wrlv6\""
Sep 06 19:20:33.440646 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:33.440557    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 06 19:20:33.441475 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:33.441440    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 06 19:20:33.501066 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:33.501033    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfs4r\" (UniqueName: \"kubernetes.io/projected/421b7405-9072-4dc4-88b4-eec6c50ddb5f-kube-api-access-nfs4r\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd\" (UID: \"421b7405-9072-4dc4-88b4-eec6c50ddb5f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd"
Sep 06 19:20:33.501066 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:33.501065    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/421b7405-9072-4dc4-88b4-eec6c50ddb5f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd\" (UID: \"421b7405-9072-4dc4-88b4-eec6c50ddb5f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd"
Sep 06 19:20:33.501232 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:33.501098    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/421b7405-9072-4dc4-88b4-eec6c50ddb5f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd\" (UID: \"421b7405-9072-4dc4-88b4-eec6c50ddb5f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd"
Sep 06 19:20:33.602203 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:33.602173    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfs4r\" (UniqueName: \"kubernetes.io/projected/421b7405-9072-4dc4-88b4-eec6c50ddb5f-kube-api-access-nfs4r\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd\" (UID: \"421b7405-9072-4dc4-88b4-eec6c50ddb5f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd"
Sep 06 19:20:33.602355 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:33.602205    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/421b7405-9072-4dc4-88b4-eec6c50ddb5f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd\" (UID: \"421b7405-9072-4dc4-88b4-eec6c50ddb5f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd"
Sep 06 19:20:33.602355 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:33.602237    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/421b7405-9072-4dc4-88b4-eec6c50ddb5f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd\" (UID: \"421b7405-9072-4dc4-88b4-eec6c50ddb5f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd"
Sep 06 19:20:33.602541 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:33.602524    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/421b7405-9072-4dc4-88b4-eec6c50ddb5f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd\" (UID: \"421b7405-9072-4dc4-88b4-eec6c50ddb5f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd"
Sep 06 19:20:33.602590 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:33.602572    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/421b7405-9072-4dc4-88b4-eec6c50ddb5f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd\" (UID: \"421b7405-9072-4dc4-88b4-eec6c50ddb5f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd"
Sep 06 19:20:33.614203 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:33.614174    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfs4r\" (UniqueName: \"kubernetes.io/projected/421b7405-9072-4dc4-88b4-eec6c50ddb5f-kube-api-access-nfs4r\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd\" (UID: \"421b7405-9072-4dc4-88b4-eec6c50ddb5f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd"
Sep 06 19:20:33.747341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:33.747312    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd"
Sep 06 19:20:34.015342 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.015320    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd"]
Sep 06 19:20:34.017406 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:20:34.017378    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421b7405_9072_4dc4_88b4_eec6c50ddb5f.slice/crio-0c7a0a76f9bc8e5695ed0e09884943e6406a792cff85cdac240e033bb42d9fa5 WatchSource:0}: Error finding container 0c7a0a76f9bc8e5695ed0e09884943e6406a792cff85cdac240e033bb42d9fa5: Status 404 returned error can't find the container with id 0c7a0a76f9bc8e5695ed0e09884943e6406a792cff85cdac240e033bb42d9fa5
Sep 06 19:20:34.375765 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.375695    2566 generic.go:356] "Generic (PLEG): container finished" podID="421b7405-9072-4dc4-88b4-eec6c50ddb5f" containerID="ff68a6c7f06387f72a069e6fec2ac66d769b4fdcb5a946ffbccd3e0c5ac3376f" exitCode=0
Sep 06 19:20:34.375912 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.375784    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd" event={"ID":"421b7405-9072-4dc4-88b4-eec6c50ddb5f","Type":"ContainerDied","Data":"ff68a6c7f06387f72a069e6fec2ac66d769b4fdcb5a946ffbccd3e0c5ac3376f"}
Sep 06 19:20:34.375912 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.375837    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd" event={"ID":"421b7405-9072-4dc4-88b4-eec6c50ddb5f","Type":"ContainerStarted","Data":"0c7a0a76f9bc8e5695ed0e09884943e6406a792cff85cdac240e033bb42d9fa5"}
Sep 06 19:20:34.531277 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.531244    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn"]
Sep 06 19:20:34.534193 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.534177    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn"
Sep 06 19:20:34.537804 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.537786    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 06 19:20:34.538586 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.538565    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b4hv4\""
Sep 06 19:20:34.538938 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.538924    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 06 19:20:34.539023 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.538926    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 06 19:20:34.542987 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.542969    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 06 19:20:34.563580 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.563553    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn"]
Sep 06 19:20:34.609859 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.609832    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e4b00d56-9a82-4e78-a700-49d155aeb7ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d4b9b44b9-wkxwn\" (UID: \"e4b00d56-9a82-4e78-a700-49d155aeb7ac\") " pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn"
Sep 06 19:20:34.609956 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.609862    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49ql8\" (UniqueName: \"kubernetes.io/projected/e4b00d56-9a82-4e78-a700-49d155aeb7ac-kube-api-access-49ql8\") pod \"opendatahub-operator-controller-manager-d4b9b44b9-wkxwn\" (UID: \"e4b00d56-9a82-4e78-a700-49d155aeb7ac\") " pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn"
Sep 06 19:20:34.609956 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.609937    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e4b00d56-9a82-4e78-a700-49d155aeb7ac-manifests\") pod \"opendatahub-operator-controller-manager-d4b9b44b9-wkxwn\" (UID: \"e4b00d56-9a82-4e78-a700-49d155aeb7ac\") " pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn"
Sep 06 19:20:34.609956 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.609953    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e4b00d56-9a82-4e78-a700-49d155aeb7ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-d4b9b44b9-wkxwn\" (UID: \"e4b00d56-9a82-4e78-a700-49d155aeb7ac\") " pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn"
Sep 06 19:20:34.610076 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.610022    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4b00d56-9a82-4e78-a700-49d155aeb7ac-tmp\") pod \"opendatahub-operator-controller-manager-d4b9b44b9-wkxwn\" (UID: \"e4b00d56-9a82-4e78-a700-49d155aeb7ac\") " pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn"
Sep 06 19:20:34.710679 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.710592    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e4b00d56-9a82-4e78-a700-49d155aeb7ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d4b9b44b9-wkxwn\" (UID: \"e4b00d56-9a82-4e78-a700-49d155aeb7ac\") " pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn"
Sep 06 19:20:34.710816 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.710690    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49ql8\" (UniqueName: \"kubernetes.io/projected/e4b00d56-9a82-4e78-a700-49d155aeb7ac-kube-api-access-49ql8\") pod \"opendatahub-operator-controller-manager-d4b9b44b9-wkxwn\" (UID: \"e4b00d56-9a82-4e78-a700-49d155aeb7ac\") " pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn"
Sep 06 19:20:34.710816 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.710790    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e4b00d56-9a82-4e78-a700-49d155aeb7ac-manifests\") pod \"opendatahub-operator-controller-manager-d4b9b44b9-wkxwn\" (UID: \"e4b00d56-9a82-4e78-a700-49d155aeb7ac\") " pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn"
Sep 06 19:20:34.710937 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.710817    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e4b00d56-9a82-4e78-a700-49d155aeb7ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-d4b9b44b9-wkxwn\" (UID: \"e4b00d56-9a82-4e78-a700-49d155aeb7ac\") " pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn"
Sep 06 19:20:34.710937 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.710877    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4b00d56-9a82-4e78-a700-49d155aeb7ac-tmp\") pod \"opendatahub-operator-controller-manager-d4b9b44b9-wkxwn\" (UID: \"e4b00d56-9a82-4e78-a700-49d155aeb7ac\") " pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn"
Sep 06 19:20:34.711190 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.711169    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e4b00d56-9a82-4e78-a700-49d155aeb7ac-manifests\") pod \"opendatahub-operator-controller-manager-d4b9b44b9-wkxwn\" (UID: \"e4b00d56-9a82-4e78-a700-49d155aeb7ac\") " pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn"
Sep 06 19:20:34.713265 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.713239    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4b00d56-9a82-4e78-a700-49d155aeb7ac-tmp\") pod \"opendatahub-operator-controller-manager-d4b9b44b9-wkxwn\" (UID: \"e4b00d56-9a82-4e78-a700-49d155aeb7ac\") " pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn"
Sep 06 19:20:34.713419 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.713405    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e4b00d56-9a82-4e78-a700-49d155aeb7ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-d4b9b44b9-wkxwn\" (UID: \"e4b00d56-9a82-4e78-a700-49d155aeb7ac\") " pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn"
Sep 06 19:20:34.713467 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.713426    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e4b00d56-9a82-4e78-a700-49d155aeb7ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d4b9b44b9-wkxwn\" (UID: \"e4b00d56-9a82-4e78-a700-49d155aeb7ac\") " pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn"
Sep 06 19:20:34.718976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.718946    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49ql8\" (UniqueName: \"kubernetes.io/projected/e4b00d56-9a82-4e78-a700-49d155aeb7ac-kube-api-access-49ql8\") pod \"opendatahub-operator-controller-manager-d4b9b44b9-wkxwn\" (UID: \"e4b00d56-9a82-4e78-a700-49d155aeb7ac\") " pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn"
Sep 06 19:20:34.844845 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:34.844817    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn"
Sep 06 19:20:35.148719 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:35.148694    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn"]
Sep 06 19:20:35.150513 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:20:35.150476    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b00d56_9a82_4e78_a700_49d155aeb7ac.slice/crio-1e1d095a56d4370e0a095b18eff22cbfe7737314acab3aa3910b14c9af062a1e WatchSource:0}: Error finding container 1e1d095a56d4370e0a095b18eff22cbfe7737314acab3aa3910b14c9af062a1e: Status 404 returned error can't find the container with id 1e1d095a56d4370e0a095b18eff22cbfe7737314acab3aa3910b14c9af062a1e
Sep 06 19:20:35.382112 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:35.382030    2566 generic.go:356] "Generic (PLEG): container finished" podID="421b7405-9072-4dc4-88b4-eec6c50ddb5f" containerID="95688ef2bff31d94d4f1bf089aaeecd033a1ccec466dce0c120d801b68ca9e3b" exitCode=0
Sep 06 19:20:35.382269 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:35.382122    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd" event={"ID":"421b7405-9072-4dc4-88b4-eec6c50ddb5f","Type":"ContainerDied","Data":"95688ef2bff31d94d4f1bf089aaeecd033a1ccec466dce0c120d801b68ca9e3b"}
Sep 06 19:20:35.383735 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:35.383584    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn" event={"ID":"e4b00d56-9a82-4e78-a700-49d155aeb7ac","Type":"ContainerStarted","Data":"1e1d095a56d4370e0a095b18eff22cbfe7737314acab3aa3910b14c9af062a1e"}
Sep 06 19:20:36.392359 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:36.392328    2566 generic.go:356] "Generic (PLEG): container finished" podID="421b7405-9072-4dc4-88b4-eec6c50ddb5f" containerID="20d2a1009a8696bd25491b59dba66e3d08341e156fb41e783a7bd558c838c87f" exitCode=0
Sep 06 19:20:36.392789 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:36.392413    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd" event={"ID":"421b7405-9072-4dc4-88b4-eec6c50ddb5f","Type":"ContainerDied","Data":"20d2a1009a8696bd25491b59dba66e3d08341e156fb41e783a7bd558c838c87f"}
Sep 06 19:20:37.528662 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.528630    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d67f7658d-94bzn"]
Sep 06 19:20:37.533459 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.533437    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:37.544984 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.544844    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d67f7658d-94bzn"]
Sep 06 19:20:37.636993 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.636956    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsjwv\" (UniqueName: \"kubernetes.io/projected/1bb3e397-280b-49f1-bb38-c6c351489c10-kube-api-access-vsjwv\") pod \"console-d67f7658d-94bzn\" (UID: \"1bb3e397-280b-49f1-bb38-c6c351489c10\") " pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:37.637152 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.637037    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bb3e397-280b-49f1-bb38-c6c351489c10-console-oauth-config\") pod \"console-d67f7658d-94bzn\" (UID: \"1bb3e397-280b-49f1-bb38-c6c351489c10\") " pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:37.637152 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.637075    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bb3e397-280b-49f1-bb38-c6c351489c10-console-config\") pod \"console-d67f7658d-94bzn\" (UID: \"1bb3e397-280b-49f1-bb38-c6c351489c10\") " pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:37.637152 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.637101    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb3e397-280b-49f1-bb38-c6c351489c10-service-ca\") pod \"console-d67f7658d-94bzn\" (UID: \"1bb3e397-280b-49f1-bb38-c6c351489c10\") " pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:37.637275 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.637153    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bb3e397-280b-49f1-bb38-c6c351489c10-trusted-ca-bundle\") pod \"console-d67f7658d-94bzn\" (UID: \"1bb3e397-280b-49f1-bb38-c6c351489c10\") " pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:37.637275 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.637228    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bb3e397-280b-49f1-bb38-c6c351489c10-console-serving-cert\") pod \"console-d67f7658d-94bzn\" (UID: \"1bb3e397-280b-49f1-bb38-c6c351489c10\") " pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:37.637275 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.637256    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bb3e397-280b-49f1-bb38-c6c351489c10-oauth-serving-cert\") pod \"console-d67f7658d-94bzn\" (UID: \"1bb3e397-280b-49f1-bb38-c6c351489c10\") " pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:37.737706 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.737671    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsjwv\" (UniqueName: \"kubernetes.io/projected/1bb3e397-280b-49f1-bb38-c6c351489c10-kube-api-access-vsjwv\") pod \"console-d67f7658d-94bzn\" (UID: \"1bb3e397-280b-49f1-bb38-c6c351489c10\") " pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:37.737894 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.737766    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bb3e397-280b-49f1-bb38-c6c351489c10-console-oauth-config\") pod \"console-d67f7658d-94bzn\" (UID: \"1bb3e397-280b-49f1-bb38-c6c351489c10\") " pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:37.737894 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.737808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bb3e397-280b-49f1-bb38-c6c351489c10-console-config\") pod \"console-d67f7658d-94bzn\" (UID: \"1bb3e397-280b-49f1-bb38-c6c351489c10\") " pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:37.737894 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.737837    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb3e397-280b-49f1-bb38-c6c351489c10-service-ca\") pod \"console-d67f7658d-94bzn\" (UID: \"1bb3e397-280b-49f1-bb38-c6c351489c10\") " pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:37.737894 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.737861    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bb3e397-280b-49f1-bb38-c6c351489c10-trusted-ca-bundle\") pod \"console-d67f7658d-94bzn\" (UID: \"1bb3e397-280b-49f1-bb38-c6c351489c10\") " pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:37.738132 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.737905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bb3e397-280b-49f1-bb38-c6c351489c10-console-serving-cert\") pod \"console-d67f7658d-94bzn\" (UID: \"1bb3e397-280b-49f1-bb38-c6c351489c10\") " pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:37.738132 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.737928    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bb3e397-280b-49f1-bb38-c6c351489c10-oauth-serving-cert\") pod \"console-d67f7658d-94bzn\" (UID: \"1bb3e397-280b-49f1-bb38-c6c351489c10\") " pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:37.738819 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.738579    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bb3e397-280b-49f1-bb38-c6c351489c10-console-config\") pod \"console-d67f7658d-94bzn\" (UID: \"1bb3e397-280b-49f1-bb38-c6c351489c10\") " pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:37.738819 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.738654    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb3e397-280b-49f1-bb38-c6c351489c10-service-ca\") pod \"console-d67f7658d-94bzn\" (UID: \"1bb3e397-280b-49f1-bb38-c6c351489c10\") " pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:37.738819 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.738775    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bb3e397-280b-49f1-bb38-c6c351489c10-oauth-serving-cert\") pod \"console-d67f7658d-94bzn\" (UID: \"1bb3e397-280b-49f1-bb38-c6c351489c10\") " pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:37.739169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.739143    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bb3e397-280b-49f1-bb38-c6c351489c10-trusted-ca-bundle\") pod \"console-d67f7658d-94bzn\" (UID: \"1bb3e397-280b-49f1-bb38-c6c351489c10\") " pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:37.741056 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.741033    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bb3e397-280b-49f1-bb38-c6c351489c10-console-serving-cert\") pod \"console-d67f7658d-94bzn\" (UID: \"1bb3e397-280b-49f1-bb38-c6c351489c10\") " pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:37.742024 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.742005    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bb3e397-280b-49f1-bb38-c6c351489c10-console-oauth-config\") pod \"console-d67f7658d-94bzn\" (UID: \"1bb3e397-280b-49f1-bb38-c6c351489c10\") " pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:37.749290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.749173    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsjwv\" (UniqueName: \"kubernetes.io/projected/1bb3e397-280b-49f1-bb38-c6c351489c10-kube-api-access-vsjwv\") pod \"console-d67f7658d-94bzn\" (UID: \"1bb3e397-280b-49f1-bb38-c6c351489c10\") " pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:37.846090 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:37.846054    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:38.044404 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:38.044380    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd"
Sep 06 19:20:38.141392 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:38.141372    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfs4r\" (UniqueName: \"kubernetes.io/projected/421b7405-9072-4dc4-88b4-eec6c50ddb5f-kube-api-access-nfs4r\") pod \"421b7405-9072-4dc4-88b4-eec6c50ddb5f\" (UID: \"421b7405-9072-4dc4-88b4-eec6c50ddb5f\") "
Sep 06 19:20:38.141480 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:38.141419    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/421b7405-9072-4dc4-88b4-eec6c50ddb5f-util\") pod \"421b7405-9072-4dc4-88b4-eec6c50ddb5f\" (UID: \"421b7405-9072-4dc4-88b4-eec6c50ddb5f\") "
Sep 06 19:20:38.141480 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:38.141435    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/421b7405-9072-4dc4-88b4-eec6c50ddb5f-bundle\") pod \"421b7405-9072-4dc4-88b4-eec6c50ddb5f\" (UID: \"421b7405-9072-4dc4-88b4-eec6c50ddb5f\") "
Sep 06 19:20:38.142177 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:38.142149    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421b7405-9072-4dc4-88b4-eec6c50ddb5f-bundle" (OuterVolumeSpecName: "bundle") pod "421b7405-9072-4dc4-88b4-eec6c50ddb5f" (UID: "421b7405-9072-4dc4-88b4-eec6c50ddb5f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:20:38.143531 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:38.143509    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421b7405-9072-4dc4-88b4-eec6c50ddb5f-kube-api-access-nfs4r" (OuterVolumeSpecName: "kube-api-access-nfs4r") pod "421b7405-9072-4dc4-88b4-eec6c50ddb5f" (UID: "421b7405-9072-4dc4-88b4-eec6c50ddb5f"). InnerVolumeSpecName "kube-api-access-nfs4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:20:38.147748 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:38.147726    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421b7405-9072-4dc4-88b4-eec6c50ddb5f-util" (OuterVolumeSpecName: "util") pod "421b7405-9072-4dc4-88b4-eec6c50ddb5f" (UID: "421b7405-9072-4dc4-88b4-eec6c50ddb5f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:20:38.242850 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:38.242826    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfs4r\" (UniqueName: \"kubernetes.io/projected/421b7405-9072-4dc4-88b4-eec6c50ddb5f-kube-api-access-nfs4r\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:20:38.242850 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:38.242847    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/421b7405-9072-4dc4-88b4-eec6c50ddb5f-util\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:20:38.242981 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:38.242858    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/421b7405-9072-4dc4-88b4-eec6c50ddb5f-bundle\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:20:38.293971 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:38.293905    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d67f7658d-94bzn"]
Sep 06 19:20:38.297211 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:20:38.297188    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb3e397_280b_49f1_bb38_c6c351489c10.slice/crio-a1bbdcb7df6a24fe58e4e4bcae74b09d2628ad766a8706a1536ec9a207b0d4bd WatchSource:0}: Error finding container a1bbdcb7df6a24fe58e4e4bcae74b09d2628ad766a8706a1536ec9a207b0d4bd: Status 404 returned error can't find the container with id a1bbdcb7df6a24fe58e4e4bcae74b09d2628ad766a8706a1536ec9a207b0d4bd
Sep 06 19:20:38.404887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:38.404852    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd" event={"ID":"421b7405-9072-4dc4-88b4-eec6c50ddb5f","Type":"ContainerDied","Data":"0c7a0a76f9bc8e5695ed0e09884943e6406a792cff85cdac240e033bb42d9fa5"}
Sep 06 19:20:38.404887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:38.404868    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d59jd"
Sep 06 19:20:38.405099 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:38.404900    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c7a0a76f9bc8e5695ed0e09884943e6406a792cff85cdac240e033bb42d9fa5"
Sep 06 19:20:38.406802 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:38.406772    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn" event={"ID":"e4b00d56-9a82-4e78-a700-49d155aeb7ac","Type":"ContainerStarted","Data":"cd8be53303471c0fccf523c532483cc1353ce87a777c88dc2cbd44ec67b27401"}
Sep 06 19:20:38.408764 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:38.408736    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d67f7658d-94bzn" event={"ID":"1bb3e397-280b-49f1-bb38-c6c351489c10","Type":"ContainerStarted","Data":"b629b902116d5704d88b1d277ff9109c02ef45357e3477884dad6f661a59c221"}
Sep 06 19:20:38.408886 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:38.408780    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d67f7658d-94bzn" event={"ID":"1bb3e397-280b-49f1-bb38-c6c351489c10","Type":"ContainerStarted","Data":"a1bbdcb7df6a24fe58e4e4bcae74b09d2628ad766a8706a1536ec9a207b0d4bd"}
Sep 06 19:20:38.452890 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:38.452847    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d67f7658d-94bzn" podStartSLOduration=1.4528296520000001 podStartE2EDuration="1.452829652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:20:38.451879486 +0000 UTC m=+572.633061292" watchObservedRunningTime="2026-09-06 19:20:38.452829652 +0000 UTC m=+572.634011459"
Sep 06 19:20:39.420419 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:39.420370    2566 generic.go:356] "Generic (PLEG): container finished" podID="e4b00d56-9a82-4e78-a700-49d155aeb7ac" containerID="cd8be53303471c0fccf523c532483cc1353ce87a777c88dc2cbd44ec67b27401" exitCode=0
Sep 06 19:20:39.420893 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:39.420431    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn" event={"ID":"e4b00d56-9a82-4e78-a700-49d155aeb7ac","Type":"ContainerDied","Data":"cd8be53303471c0fccf523c532483cc1353ce87a777c88dc2cbd44ec67b27401"}
Sep 06 19:20:40.426903 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:40.426866    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn" event={"ID":"e4b00d56-9a82-4e78-a700-49d155aeb7ac","Type":"ContainerStarted","Data":"33c044eab92b28788ad25ad8df384876783ea5b0c405fc72eb3ef13ff7d1a1fc"}
Sep 06 19:20:40.427373 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:40.426978    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn"
Sep 06 19:20:40.447388 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:40.447346    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn" podStartSLOduration=2.095925035 podStartE2EDuration="6.447330982s" totalImagesPullingTime="3.336937601s" totalInitContainerRuntime="1.014468346s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:20:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:20:40.445434911 +0000 UTC m=+574.626616715" watchObservedRunningTime="2026-09-06 19:20:40.447330982 +0000 UTC m=+574.628512845"
Sep 06 19:20:47.846724 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:47.846689    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:47.846724 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:47.846764    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:47.851449 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:47.851428    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:48.468384 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:48.468355    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d67f7658d-94bzn"
Sep 06 19:20:48.511907 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:48.511872    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f65f49f84-d62pt"]
Sep 06 19:20:51.434473 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:51.434443    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d4b9b44b9-wkxwn"
Sep 06 19:20:54.435266 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.435233    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-8b7764445-x6bd7"]
Sep 06 19:20:54.435625 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.435548    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="421b7405-9072-4dc4-88b4-eec6c50ddb5f" containerName="extract"
Sep 06 19:20:54.435625 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.435560    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="421b7405-9072-4dc4-88b4-eec6c50ddb5f" containerName="extract"
Sep 06 19:20:54.435625 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.435574    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="421b7405-9072-4dc4-88b4-eec6c50ddb5f" containerName="util"
Sep 06 19:20:54.435625 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.435579    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="421b7405-9072-4dc4-88b4-eec6c50ddb5f" containerName="util"
Sep 06 19:20:54.435625 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.435585    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="421b7405-9072-4dc4-88b4-eec6c50ddb5f" containerName="pull"
Sep 06 19:20:54.435625 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.435591    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="421b7405-9072-4dc4-88b4-eec6c50ddb5f" containerName="pull"
Sep 06 19:20:54.435818 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.435676    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="421b7405-9072-4dc4-88b4-eec6c50ddb5f" containerName="extract"
Sep 06 19:20:54.443536 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.443515    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8b7764445-x6bd7"
Sep 06 19:20:54.445943 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.445923    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 06 19:20:54.446056 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.445986    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 06 19:20:54.446119 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.446060    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 06 19:20:54.446119 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.446060    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 06 19:20:54.447678 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.447657    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-b7pgb\""
Sep 06 19:20:54.450467 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.450436    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8b7764445-x6bd7"]
Sep 06 19:20:54.576412 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.576376    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78ee1fa5-948c-4a27-a4a3-6934d802ed1b-tls-certs\") pod \"kube-auth-proxy-8b7764445-x6bd7\" (UID: \"78ee1fa5-948c-4a27-a4a3-6934d802ed1b\") " pod="openshift-ingress/kube-auth-proxy-8b7764445-x6bd7"
Sep 06 19:20:54.576412 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.576407    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjt8t\" (UniqueName: \"kubernetes.io/projected/78ee1fa5-948c-4a27-a4a3-6934d802ed1b-kube-api-access-qjt8t\") pod \"kube-auth-proxy-8b7764445-x6bd7\" (UID: \"78ee1fa5-948c-4a27-a4a3-6934d802ed1b\") " pod="openshift-ingress/kube-auth-proxy-8b7764445-x6bd7"
Sep 06 19:20:54.576623 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.576494    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78ee1fa5-948c-4a27-a4a3-6934d802ed1b-tmp\") pod \"kube-auth-proxy-8b7764445-x6bd7\" (UID: \"78ee1fa5-948c-4a27-a4a3-6934d802ed1b\") " pod="openshift-ingress/kube-auth-proxy-8b7764445-x6bd7"
Sep 06 19:20:54.677773 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.677745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78ee1fa5-948c-4a27-a4a3-6934d802ed1b-tls-certs\") pod \"kube-auth-proxy-8b7764445-x6bd7\" (UID: \"78ee1fa5-948c-4a27-a4a3-6934d802ed1b\") " pod="openshift-ingress/kube-auth-proxy-8b7764445-x6bd7"
Sep 06 19:20:54.677956 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.677783    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjt8t\" (UniqueName: \"kubernetes.io/projected/78ee1fa5-948c-4a27-a4a3-6934d802ed1b-kube-api-access-qjt8t\") pod \"kube-auth-proxy-8b7764445-x6bd7\" (UID: \"78ee1fa5-948c-4a27-a4a3-6934d802ed1b\") " pod="openshift-ingress/kube-auth-proxy-8b7764445-x6bd7"
Sep 06 19:20:54.677956 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.677836    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78ee1fa5-948c-4a27-a4a3-6934d802ed1b-tmp\") pod \"kube-auth-proxy-8b7764445-x6bd7\" (UID: \"78ee1fa5-948c-4a27-a4a3-6934d802ed1b\") " pod="openshift-ingress/kube-auth-proxy-8b7764445-x6bd7"
Sep 06 19:20:54.680167 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.680143    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78ee1fa5-948c-4a27-a4a3-6934d802ed1b-tmp\") pod \"kube-auth-proxy-8b7764445-x6bd7\" (UID: \"78ee1fa5-948c-4a27-a4a3-6934d802ed1b\") " pod="openshift-ingress/kube-auth-proxy-8b7764445-x6bd7"
Sep 06 19:20:54.680333 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.680312    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78ee1fa5-948c-4a27-a4a3-6934d802ed1b-tls-certs\") pod \"kube-auth-proxy-8b7764445-x6bd7\" (UID: \"78ee1fa5-948c-4a27-a4a3-6934d802ed1b\") " pod="openshift-ingress/kube-auth-proxy-8b7764445-x6bd7"
Sep 06 19:20:54.689735 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.689680    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjt8t\" (UniqueName: \"kubernetes.io/projected/78ee1fa5-948c-4a27-a4a3-6934d802ed1b-kube-api-access-qjt8t\") pod \"kube-auth-proxy-8b7764445-x6bd7\" (UID: \"78ee1fa5-948c-4a27-a4a3-6934d802ed1b\") " pod="openshift-ingress/kube-auth-proxy-8b7764445-x6bd7"
Sep 06 19:20:54.754829 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:54.754801    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8b7764445-x6bd7"
Sep 06 19:20:55.048564 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.048542    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8b7764445-x6bd7"]
Sep 06 19:20:55.050174 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:20:55.050150    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ee1fa5_948c_4a27_a4a3_6934d802ed1b.slice/crio-cb9874c578016452f5a4aa19b183f93ed238723b09fed069147f053ba6e1821b WatchSource:0}: Error finding container cb9874c578016452f5a4aa19b183f93ed238723b09fed069147f053ba6e1821b: Status 404 returned error can't find the container with id cb9874c578016452f5a4aa19b183f93ed238723b09fed069147f053ba6e1821b
Sep 06 19:20:55.498738 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.498704    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8b7764445-x6bd7" event={"ID":"78ee1fa5-948c-4a27-a4a3-6934d802ed1b","Type":"ContainerStarted","Data":"cb9874c578016452f5a4aa19b183f93ed238723b09fed069147f053ba6e1821b"}
Sep 06 19:20:55.688880 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.688853    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"]
Sep 06 19:20:55.693664 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.693647    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:20:55.696096 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.696072    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 06 19:20:55.696096 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.696089    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 06 19:20:55.696226 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.696144    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-msdgd\""
Sep 06 19:20:55.696512 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.696498    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 06 19:20:55.696986 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.696972    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 06 19:20:55.706813 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.706785    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"]
Sep 06 19:20:55.787400 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.787320    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-slkpx\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:20:55.787400 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.787378    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-slkpx\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:20:55.787626 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.787412    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-slkpx\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:20:55.787626 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.787522    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-slkpx\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:20:55.787626 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.787554    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-slkpx\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:20:55.787626 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.787583    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss9tk\" (UniqueName: \"kubernetes.io/projected/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-kube-api-access-ss9tk\") pod \"istiod-openshift-gateway-7b7995d5cd-slkpx\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:20:55.787841 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.787644    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-slkpx\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:20:55.888555 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.888512    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-slkpx\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:20:55.888757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.888574    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-slkpx\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:20:55.888757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.888622    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-slkpx\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:20:55.888757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.888716    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-slkpx\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:20:55.888757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.888743    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-slkpx\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:20:55.888922 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.888777    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss9tk\" (UniqueName: \"kubernetes.io/projected/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-kube-api-access-ss9tk\") pod \"istiod-openshift-gateway-7b7995d5cd-slkpx\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:20:55.888922 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.888818    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-slkpx\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:20:55.890182 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.889314    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-slkpx\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:20:55.892045 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.892005    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-slkpx\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:20:55.892045 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.892025    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-slkpx\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:20:55.892321 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.892289    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-slkpx\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:20:55.892672 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.892635    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-slkpx\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:20:55.897437 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.897416    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss9tk\" (UniqueName: \"kubernetes.io/projected/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-kube-api-access-ss9tk\") pod \"istiod-openshift-gateway-7b7995d5cd-slkpx\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:20:55.897552 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:55.897530    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-slkpx\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:20:56.003690 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:56.003648    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:20:56.287119 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:56.287090    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"]
Sep 06 19:20:56.289648 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:20:56.289587    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda317ba05_0ec8_4f70_aa2d_8d7830dcd4a5.slice/crio-7ddd8d304605f7aa0d1a89b00bdaf23793f1d646af1a492443cc4bc4aff64326 WatchSource:0}: Error finding container 7ddd8d304605f7aa0d1a89b00bdaf23793f1d646af1a492443cc4bc4aff64326: Status 404 returned error can't find the container with id 7ddd8d304605f7aa0d1a89b00bdaf23793f1d646af1a492443cc4bc4aff64326
Sep 06 19:20:56.508447 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:56.508401    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx" event={"ID":"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5","Type":"ContainerStarted","Data":"7ddd8d304605f7aa0d1a89b00bdaf23793f1d646af1a492443cc4bc4aff64326"}
Sep 06 19:20:58.521287 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:58.521249    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8b7764445-x6bd7" event={"ID":"78ee1fa5-948c-4a27-a4a3-6934d802ed1b","Type":"ContainerStarted","Data":"748b30ff0918bd8e0a5a8bbdab368b9a03e8d19f547f59d93339d451b23e1ae9"}
Sep 06 19:20:58.536522 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:58.536478    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-8b7764445-x6bd7" podStartSLOduration=1.556945977 podStartE2EDuration="4.536464091s" totalImagesPullingTime="2.979518114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:20:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:20:58.535341493 +0000 UTC m=+592.716523299" watchObservedRunningTime="2026-09-06 19:20:58.536464091 +0000 UTC m=+592.717645898"
Sep 06 19:20:59.841034 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:59.840995    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 06 19:20:59.841401 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:20:59.841081    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 06 19:21:00.533181 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:00.533145    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx" event={"ID":"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5","Type":"ContainerStarted","Data":"75f1992b1b24caaad0875b96c9a7cccf1c72ee911dcbe3b5a5250a666a888c38"}
Sep 06 19:21:00.533349 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:00.533212    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:21:00.534975 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:00.534952    2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-slkpx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 06 19:21:00.535059 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:00.534998    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx" podUID="a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 06 19:21:00.552113 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:00.552065    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx" podStartSLOduration=2.003219263 podStartE2EDuration="5.552046255s" totalImagesPullingTime="3.548826992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:20:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:21:00.550366126 +0000 UTC m=+594.731547933" watchObservedRunningTime="2026-09-06 19:21:00.552046255 +0000 UTC m=+594.733228063"
Sep 06 19:21:01.538453 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:01.538424    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:21:06.300099 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:06.300066    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnssq_9254e6f1-b73d-49cc-8bc1-513925ccde4f/ovn-acl-logging/0.log"
Sep 06 19:21:06.300569 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:06.300550    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnssq_9254e6f1-b73d-49cc-8bc1-513925ccde4f/ovn-acl-logging/0.log"
Sep 06 19:21:12.485656 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:12.485619    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c4c9c58b7-hfv74"]
Sep 06 19:21:12.489204 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:12.489182    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c4c9c58b7-hfv74"
Sep 06 19:21:12.491573 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:12.491552    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 06 19:21:12.492654 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:12.492619    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 06 19:21:12.492654 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:12.492628    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 06 19:21:12.492654 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:12.492652    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 06 19:21:12.492883 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:12.492651    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 06 19:21:12.492883 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:12.492745    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ldlmh\""
Sep 06 19:21:12.499216 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:12.499195    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c4c9c58b7-hfv74"]
Sep 06 19:21:12.531408 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:12.531314    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg74z\" (UniqueName: \"kubernetes.io/projected/b98aafd2-cccd-41a8-abce-c8ee82e5a972-kube-api-access-bg74z\") pod \"lws-controller-manager-5c4c9c58b7-hfv74\" (UID: \"b98aafd2-cccd-41a8-abce-c8ee82e5a972\") " pod="openshift-lws-operator/lws-controller-manager-5c4c9c58b7-hfv74"
Sep 06 19:21:12.531518 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:12.531420    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b98aafd2-cccd-41a8-abce-c8ee82e5a972-cert\") pod \"lws-controller-manager-5c4c9c58b7-hfv74\" (UID: \"b98aafd2-cccd-41a8-abce-c8ee82e5a972\") " pod="openshift-lws-operator/lws-controller-manager-5c4c9c58b7-hfv74"
Sep 06 19:21:12.531518 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:12.531472    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b98aafd2-cccd-41a8-abce-c8ee82e5a972-metrics-cert\") pod \"lws-controller-manager-5c4c9c58b7-hfv74\" (UID: \"b98aafd2-cccd-41a8-abce-c8ee82e5a972\") " pod="openshift-lws-operator/lws-controller-manager-5c4c9c58b7-hfv74"
Sep 06 19:21:12.531518 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:12.531509    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b98aafd2-cccd-41a8-abce-c8ee82e5a972-manager-config\") pod \"lws-controller-manager-5c4c9c58b7-hfv74\" (UID: \"b98aafd2-cccd-41a8-abce-c8ee82e5a972\") " pod="openshift-lws-operator/lws-controller-manager-5c4c9c58b7-hfv74"
Sep 06 19:21:12.632238 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:12.632210    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b98aafd2-cccd-41a8-abce-c8ee82e5a972-cert\") pod \"lws-controller-manager-5c4c9c58b7-hfv74\" (UID: \"b98aafd2-cccd-41a8-abce-c8ee82e5a972\") " pod="openshift-lws-operator/lws-controller-manager-5c4c9c58b7-hfv74"
Sep 06 19:21:12.632361 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:12.632251    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b98aafd2-cccd-41a8-abce-c8ee82e5a972-metrics-cert\") pod \"lws-controller-manager-5c4c9c58b7-hfv74\" (UID: \"b98aafd2-cccd-41a8-abce-c8ee82e5a972\") " pod="openshift-lws-operator/lws-controller-manager-5c4c9c58b7-hfv74"
Sep 06 19:21:12.632361 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:12.632274    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b98aafd2-cccd-41a8-abce-c8ee82e5a972-manager-config\") pod \"lws-controller-manager-5c4c9c58b7-hfv74\" (UID: \"b98aafd2-cccd-41a8-abce-c8ee82e5a972\") " pod="openshift-lws-operator/lws-controller-manager-5c4c9c58b7-hfv74"
Sep 06 19:21:12.632361 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:12.632319    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg74z\" (UniqueName: \"kubernetes.io/projected/b98aafd2-cccd-41a8-abce-c8ee82e5a972-kube-api-access-bg74z\") pod \"lws-controller-manager-5c4c9c58b7-hfv74\" (UID: \"b98aafd2-cccd-41a8-abce-c8ee82e5a972\") " pod="openshift-lws-operator/lws-controller-manager-5c4c9c58b7-hfv74"
Sep 06 19:21:12.633147 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:12.633091    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b98aafd2-cccd-41a8-abce-c8ee82e5a972-manager-config\") pod \"lws-controller-manager-5c4c9c58b7-hfv74\" (UID: \"b98aafd2-cccd-41a8-abce-c8ee82e5a972\") " pod="openshift-lws-operator/lws-controller-manager-5c4c9c58b7-hfv74"
Sep 06 19:21:12.634915 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:12.634895    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b98aafd2-cccd-41a8-abce-c8ee82e5a972-cert\") pod \"lws-controller-manager-5c4c9c58b7-hfv74\" (UID: \"b98aafd2-cccd-41a8-abce-c8ee82e5a972\") " pod="openshift-lws-operator/lws-controller-manager-5c4c9c58b7-hfv74"
Sep 06 19:21:12.634996 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:12.634963    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b98aafd2-cccd-41a8-abce-c8ee82e5a972-metrics-cert\") pod \"lws-controller-manager-5c4c9c58b7-hfv74\" (UID: \"b98aafd2-cccd-41a8-abce-c8ee82e5a972\") " pod="openshift-lws-operator/lws-controller-manager-5c4c9c58b7-hfv74"
Sep 06 19:21:12.641196 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:12.641177    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg74z\" (UniqueName: \"kubernetes.io/projected/b98aafd2-cccd-41a8-abce-c8ee82e5a972-kube-api-access-bg74z\") pod \"lws-controller-manager-5c4c9c58b7-hfv74\" (UID: \"b98aafd2-cccd-41a8-abce-c8ee82e5a972\") " pod="openshift-lws-operator/lws-controller-manager-5c4c9c58b7-hfv74"
Sep 06 19:21:12.800433 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:12.800357    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c4c9c58b7-hfv74"
Sep 06 19:21:13.073756 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.073728    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c4c9c58b7-hfv74"]
Sep 06 19:21:13.075571 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:21:13.075546    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98aafd2_cccd_41a8_abce_c8ee82e5a972.slice/crio-8571582d090dd30401aa02fd5a56edd32473c294e28ed0ffae4ac4fad3fa8aaa WatchSource:0}: Error finding container 8571582d090dd30401aa02fd5a56edd32473c294e28ed0ffae4ac4fad3fa8aaa: Status 404 returned error can't find the container with id 8571582d090dd30401aa02fd5a56edd32473c294e28ed0ffae4ac4fad3fa8aaa
Sep 06 19:21:13.532019 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.531963    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f65f49f84-d62pt" podUID="04714b33-d678-4645-bc98-409546ff354a" containerName="console" containerID="cri-o://8e0ba26bcdb74cd6089a20b43934b7f74b3fdbf768938cd239ab083c99e0731d" gracePeriod=15
Sep 06 19:21:13.595533 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.595502    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c4c9c58b7-hfv74" event={"ID":"b98aafd2-cccd-41a8-abce-c8ee82e5a972","Type":"ContainerStarted","Data":"8571582d090dd30401aa02fd5a56edd32473c294e28ed0ffae4ac4fad3fa8aaa"}
Sep 06 19:21:13.777373 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.777350    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:21:13.843294 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.843224    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04714b33-d678-4645-bc98-409546ff354a-console-oauth-config\") pod \"04714b33-d678-4645-bc98-409546ff354a\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") "
Sep 06 19:21:13.843294 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.843263    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-trusted-ca-bundle\") pod \"04714b33-d678-4645-bc98-409546ff354a\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") "
Sep 06 19:21:13.843488 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.843321    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwbfw\" (UniqueName: \"kubernetes.io/projected/04714b33-d678-4645-bc98-409546ff354a-kube-api-access-zwbfw\") pod \"04714b33-d678-4645-bc98-409546ff354a\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") "
Sep 06 19:21:13.843488 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.843372    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-console-config\") pod \"04714b33-d678-4645-bc98-409546ff354a\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") "
Sep 06 19:21:13.843488 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.843416    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04714b33-d678-4645-bc98-409546ff354a-console-serving-cert\") pod \"04714b33-d678-4645-bc98-409546ff354a\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") "
Sep 06 19:21:13.843488 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.843442    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-oauth-serving-cert\") pod \"04714b33-d678-4645-bc98-409546ff354a\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") "
Sep 06 19:21:13.843488 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.843479    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-service-ca\") pod \"04714b33-d678-4645-bc98-409546ff354a\" (UID: \"04714b33-d678-4645-bc98-409546ff354a\") "
Sep 06 19:21:13.843863 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.843808    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "04714b33-d678-4645-bc98-409546ff354a" (UID: "04714b33-d678-4645-bc98-409546ff354a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:21:13.843863 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.843817    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-console-config" (OuterVolumeSpecName: "console-config") pod "04714b33-d678-4645-bc98-409546ff354a" (UID: "04714b33-d678-4645-bc98-409546ff354a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:21:13.843863 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.843841    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "04714b33-d678-4645-bc98-409546ff354a" (UID: "04714b33-d678-4645-bc98-409546ff354a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:21:13.844341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.844125    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-service-ca" (OuterVolumeSpecName: "service-ca") pod "04714b33-d678-4645-bc98-409546ff354a" (UID: "04714b33-d678-4645-bc98-409546ff354a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:21:13.845543 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.845516    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04714b33-d678-4645-bc98-409546ff354a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "04714b33-d678-4645-bc98-409546ff354a" (UID: "04714b33-d678-4645-bc98-409546ff354a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:21:13.845651 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.845633    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04714b33-d678-4645-bc98-409546ff354a-kube-api-access-zwbfw" (OuterVolumeSpecName: "kube-api-access-zwbfw") pod "04714b33-d678-4645-bc98-409546ff354a" (UID: "04714b33-d678-4645-bc98-409546ff354a"). InnerVolumeSpecName "kube-api-access-zwbfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:21:13.845740 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.845721    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04714b33-d678-4645-bc98-409546ff354a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "04714b33-d678-4645-bc98-409546ff354a" (UID: "04714b33-d678-4645-bc98-409546ff354a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:21:13.944352 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.944327    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-service-ca\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:21:13.944352 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.944350    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04714b33-d678-4645-bc98-409546ff354a-console-oauth-config\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:21:13.944481 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.944359    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-trusted-ca-bundle\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:21:13.944481 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.944368    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwbfw\" (UniqueName: \"kubernetes.io/projected/04714b33-d678-4645-bc98-409546ff354a-kube-api-access-zwbfw\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:21:13.944481 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.944377    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-console-config\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:21:13.944481 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.944385    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04714b33-d678-4645-bc98-409546ff354a-console-serving-cert\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:21:13.944481 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:13.944394    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04714b33-d678-4645-bc98-409546ff354a-oauth-serving-cert\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:21:14.603053 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:14.602957    2566 generic.go:356] "Generic (PLEG): container finished" podID="04714b33-d678-4645-bc98-409546ff354a" containerID="8e0ba26bcdb74cd6089a20b43934b7f74b3fdbf768938cd239ab083c99e0731d" exitCode=0
Sep 06 19:21:14.603053 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:14.603017    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f65f49f84-d62pt"
Sep 06 19:21:14.603053 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:14.603032    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f65f49f84-d62pt" event={"ID":"04714b33-d678-4645-bc98-409546ff354a","Type":"ContainerDied","Data":"8e0ba26bcdb74cd6089a20b43934b7f74b3fdbf768938cd239ab083c99e0731d"}
Sep 06 19:21:14.603568 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:14.603072    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f65f49f84-d62pt" event={"ID":"04714b33-d678-4645-bc98-409546ff354a","Type":"ContainerDied","Data":"14a628c3999e1a970de6b040aa28dc4091674e5a1baf3a6e52ceda79f0335436"}
Sep 06 19:21:14.603568 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:14.603091    2566 scope.go:117] "RemoveContainer" containerID="8e0ba26bcdb74cd6089a20b43934b7f74b3fdbf768938cd239ab083c99e0731d"
Sep 06 19:21:14.614492 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:14.614465    2566 scope.go:117] "RemoveContainer" containerID="8e0ba26bcdb74cd6089a20b43934b7f74b3fdbf768938cd239ab083c99e0731d"
Sep 06 19:21:14.614872 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:21:14.614847    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e0ba26bcdb74cd6089a20b43934b7f74b3fdbf768938cd239ab083c99e0731d\": container with ID starting with 8e0ba26bcdb74cd6089a20b43934b7f74b3fdbf768938cd239ab083c99e0731d not found: ID does not exist" containerID="8e0ba26bcdb74cd6089a20b43934b7f74b3fdbf768938cd239ab083c99e0731d"
Sep 06 19:21:14.614955 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:14.614882    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e0ba26bcdb74cd6089a20b43934b7f74b3fdbf768938cd239ab083c99e0731d"} err="failed to get container status \"8e0ba26bcdb74cd6089a20b43934b7f74b3fdbf768938cd239ab083c99e0731d\": rpc error: code = NotFound desc = could not find container \"8e0ba26bcdb74cd6089a20b43934b7f74b3fdbf768938cd239ab083c99e0731d\": container with ID starting with 8e0ba26bcdb74cd6089a20b43934b7f74b3fdbf768938cd239ab083c99e0731d not found: ID does not exist"
Sep 06 19:21:14.624989 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:14.624963    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f65f49f84-d62pt"]
Sep 06 19:21:14.630864 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:14.630835    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f65f49f84-d62pt"]
Sep 06 19:21:16.310492 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:16.310456    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04714b33-d678-4645-bc98-409546ff354a" path="/var/lib/kubelet/pods/04714b33-d678-4645-bc98-409546ff354a/volumes"
Sep 06 19:21:17.627301 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:17.627259    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c4c9c58b7-hfv74" event={"ID":"b98aafd2-cccd-41a8-abce-c8ee82e5a972","Type":"ContainerStarted","Data":"68dec6efca48f57374fdd71ef9c7896f1ee885bdff29875f521362f8ced72ed6"}
Sep 06 19:21:17.627687 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:17.627324    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5c4c9c58b7-hfv74"
Sep 06 19:21:17.646838 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:17.646792    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5c4c9c58b7-hfv74" podStartSLOduration=1.978570253 podStartE2EDuration="5.646778449s" totalImagesPullingTime="3.668208196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:21:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:21:17.644826373 +0000 UTC m=+611.826008180" watchObservedRunningTime="2026-09-06 19:21:17.646778449 +0000 UTC m=+611.827960254"
Sep 06 19:21:18.633114 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:18.633084    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"]
Sep 06 19:21:18.633533 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:18.633280    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx" podUID="a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5" containerName="discovery" containerID="cri-o://75f1992b1b24caaad0875b96c9a7cccf1c72ee911dcbe3b5a5250a666a888c38" gracePeriod=30
Sep 06 19:21:18.879516 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:18.879495    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:21:18.995121 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:18.995090    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-kubeconfig\") pod \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") "
Sep 06 19:21:18.995276 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:18.995139    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-cacerts\") pod \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") "
Sep 06 19:21:18.995276 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:18.995192    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-token\") pod \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") "
Sep 06 19:21:18.995276 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:18.995221    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-local-certs\") pod \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") "
Sep 06 19:21:18.995276 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:18.995248    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ss9tk\" (UniqueName: \"kubernetes.io/projected/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-kube-api-access-ss9tk\") pod \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") "
Sep 06 19:21:18.995458 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:18.995303    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-csr-ca-configmap\") pod \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") "
Sep 06 19:21:18.995458 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:18.995332    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-csr-dns-cert\") pod \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\" (UID: \"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5\") "
Sep 06 19:21:18.995789 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:18.995754    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5" (UID: "a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:21:18.997848 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:18.997821    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-local-certs" (OuterVolumeSpecName: "local-certs") pod "a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5" (UID: "a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:21:18.997848 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:18.997827    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-cacerts" (OuterVolumeSpecName: "cacerts") pod "a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5" (UID: "a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:21:18.998110 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:18.998082    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-token" (OuterVolumeSpecName: "istio-token") pod "a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5" (UID: "a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:21:18.998210 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:18.998139    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5" (UID: "a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:21:18.998210 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:18.998143    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-kube-api-access-ss9tk" (OuterVolumeSpecName: "kube-api-access-ss9tk") pod "a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5" (UID: "a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5"). InnerVolumeSpecName "kube-api-access-ss9tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:21:18.998210 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:18.998156    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5" (UID: "a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:21:19.096747 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:19.096717    2566 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-kubeconfig\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:21:19.096747 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:19.096740    2566 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-cacerts\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:21:19.096747 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:19.096749    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-token\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:21:19.096924 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:19.096759    2566 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-local-certs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:21:19.096924 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:19.096767    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss9tk\" (UniqueName: \"kubernetes.io/projected/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-kube-api-access-ss9tk\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:21:19.096924 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:19.096775    2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-csr-ca-configmap\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:21:19.096924 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:19.096783    2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5-istio-csr-dns-cert\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:21:19.639107 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:19.639071    2566 generic.go:356] "Generic (PLEG): container finished" podID="a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5" containerID="75f1992b1b24caaad0875b96c9a7cccf1c72ee911dcbe3b5a5250a666a888c38" exitCode=0
Sep 06 19:21:19.639546 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:19.639117    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"
Sep 06 19:21:19.639546 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:19.639162    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx" event={"ID":"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5","Type":"ContainerDied","Data":"75f1992b1b24caaad0875b96c9a7cccf1c72ee911dcbe3b5a5250a666a888c38"}
Sep 06 19:21:19.639546 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:19.639220    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx" event={"ID":"a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5","Type":"ContainerDied","Data":"7ddd8d304605f7aa0d1a89b00bdaf23793f1d646af1a492443cc4bc4aff64326"}
Sep 06 19:21:19.639546 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:19.639251    2566 scope.go:117] "RemoveContainer" containerID="75f1992b1b24caaad0875b96c9a7cccf1c72ee911dcbe3b5a5250a666a888c38"
Sep 06 19:21:19.648362 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:19.648347    2566 scope.go:117] "RemoveContainer" containerID="75f1992b1b24caaad0875b96c9a7cccf1c72ee911dcbe3b5a5250a666a888c38"
Sep 06 19:21:19.648653 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:21:19.648631    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75f1992b1b24caaad0875b96c9a7cccf1c72ee911dcbe3b5a5250a666a888c38\": container with ID starting with 75f1992b1b24caaad0875b96c9a7cccf1c72ee911dcbe3b5a5250a666a888c38 not found: ID does not exist" containerID="75f1992b1b24caaad0875b96c9a7cccf1c72ee911dcbe3b5a5250a666a888c38"
Sep 06 19:21:19.648710 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:19.648661    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75f1992b1b24caaad0875b96c9a7cccf1c72ee911dcbe3b5a5250a666a888c38"} err="failed to get container status \"75f1992b1b24caaad0875b96c9a7cccf1c72ee911dcbe3b5a5250a666a888c38\": rpc error: code = NotFound desc = could not find container \"75f1992b1b24caaad0875b96c9a7cccf1c72ee911dcbe3b5a5250a666a888c38\": container with ID starting with 75f1992b1b24caaad0875b96c9a7cccf1c72ee911dcbe3b5a5250a666a888c38 not found: ID does not exist"
Sep 06 19:21:19.662643 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:19.662597    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"]
Sep 06 19:21:19.666588 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:19.666568    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slkpx"]
Sep 06 19:21:20.315398 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:20.315365    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5" path="/var/lib/kubelet/pods/a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5/volumes"
Sep 06 19:21:28.635107 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:21:28.635032    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5c4c9c58b7-hfv74"
Sep 06 19:22:09.095429 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.095394    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf"]
Sep 06 19:22:09.095847 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.095765    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04714b33-d678-4645-bc98-409546ff354a" containerName="console"
Sep 06 19:22:09.095847 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.095780    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="04714b33-d678-4645-bc98-409546ff354a" containerName="console"
Sep 06 19:22:09.095847 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.095796    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5" containerName="discovery"
Sep 06 19:22:09.095847 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.095802    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5" containerName="discovery"
Sep 06 19:22:09.095992 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.095874    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="04714b33-d678-4645-bc98-409546ff354a" containerName="console"
Sep 06 19:22:09.095992 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.095884    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a317ba05-0ec8-4f70-aa2d-8d7830dcd4a5" containerName="discovery"
Sep 06 19:22:09.100361 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.100340    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf"
Sep 06 19:22:09.102710 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.102686    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 06 19:22:09.102806 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.102741    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 06 19:22:09.102917 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.102902    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wrlv6\""
Sep 06 19:22:09.106651 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.106630    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf"]
Sep 06 19:22:09.195107 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.195079    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9"]
Sep 06 19:22:09.198401 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.198387    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9"
Sep 06 19:22:09.198724 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.198705    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf\" (UID: \"6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf"
Sep 06 19:22:09.198776 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.198755    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cfj6\" (UniqueName: \"kubernetes.io/projected/6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba-kube-api-access-8cfj6\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf\" (UID: \"6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf"
Sep 06 19:22:09.198813 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.198776    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf\" (UID: \"6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf"
Sep 06 19:22:09.206665 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.206645    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9"]
Sep 06 19:22:09.293803 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.293779    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k"]
Sep 06 19:22:09.297115 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.297101    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k"
Sep 06 19:22:09.299323 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.299302    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6d3f50a-003b-4018-915a-ed1f71f1c647-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9\" (UID: \"c6d3f50a-003b-4018-915a-ed1f71f1c647\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9"
Sep 06 19:22:09.299416 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.299348    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf\" (UID: \"6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf"
Sep 06 19:22:09.299456 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.299417    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6d3f50a-003b-4018-915a-ed1f71f1c647-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9\" (UID: \"c6d3f50a-003b-4018-915a-ed1f71f1c647\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9"
Sep 06 19:22:09.299456 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.299448    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cfj6\" (UniqueName: \"kubernetes.io/projected/6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba-kube-api-access-8cfj6\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf\" (UID: \"6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf"
Sep 06 19:22:09.299531 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.299465    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcg82\" (UniqueName: \"kubernetes.io/projected/c6d3f50a-003b-4018-915a-ed1f71f1c647-kube-api-access-hcg82\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9\" (UID: \"c6d3f50a-003b-4018-915a-ed1f71f1c647\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9"
Sep 06 19:22:09.299531 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.299484    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf\" (UID: \"6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf"
Sep 06 19:22:09.299729 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.299713    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf\" (UID: \"6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf"
Sep 06 19:22:09.299811 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.299797    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf\" (UID: \"6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf"
Sep 06 19:22:09.304531 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.304416    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k"]
Sep 06 19:22:09.307380 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.307357    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cfj6\" (UniqueName: \"kubernetes.io/projected/6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba-kube-api-access-8cfj6\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf\" (UID: \"6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf"
Sep 06 19:22:09.395354 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.395294    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr"]
Sep 06 19:22:09.399077 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.399062    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr"
Sep 06 19:22:09.400090 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.400071    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9fc87590-9575-4698-94ac-ae6804b13a1b-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k\" (UID: \"9fc87590-9575-4698-94ac-ae6804b13a1b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k"
Sep 06 19:22:09.400155 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.400134    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6d3f50a-003b-4018-915a-ed1f71f1c647-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9\" (UID: \"c6d3f50a-003b-4018-915a-ed1f71f1c647\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9"
Sep 06 19:22:09.400212 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.400164    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcg82\" (UniqueName: \"kubernetes.io/projected/c6d3f50a-003b-4018-915a-ed1f71f1c647-kube-api-access-hcg82\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9\" (UID: \"c6d3f50a-003b-4018-915a-ed1f71f1c647\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9"
Sep 06 19:22:09.400212 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.400193    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl7hr\" (UniqueName: \"kubernetes.io/projected/9fc87590-9575-4698-94ac-ae6804b13a1b-kube-api-access-jl7hr\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k\" (UID: \"9fc87590-9575-4698-94ac-ae6804b13a1b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k"
Sep 06 19:22:09.400317 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.400231    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6d3f50a-003b-4018-915a-ed1f71f1c647-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9\" (UID: \"c6d3f50a-003b-4018-915a-ed1f71f1c647\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9"
Sep 06 19:22:09.400317 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.400257    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9fc87590-9575-4698-94ac-ae6804b13a1b-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k\" (UID: \"9fc87590-9575-4698-94ac-ae6804b13a1b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k"
Sep 06 19:22:09.400487 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.400468    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6d3f50a-003b-4018-915a-ed1f71f1c647-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9\" (UID: \"c6d3f50a-003b-4018-915a-ed1f71f1c647\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9"
Sep 06 19:22:09.400545 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.400479    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6d3f50a-003b-4018-915a-ed1f71f1c647-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9\" (UID: \"c6d3f50a-003b-4018-915a-ed1f71f1c647\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9"
Sep 06 19:22:09.406061 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.406041    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr"]
Sep 06 19:22:09.408346 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.408320    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcg82\" (UniqueName: \"kubernetes.io/projected/c6d3f50a-003b-4018-915a-ed1f71f1c647-kube-api-access-hcg82\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9\" (UID: \"c6d3f50a-003b-4018-915a-ed1f71f1c647\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9"
Sep 06 19:22:09.411131 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.411110    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf"
Sep 06 19:22:09.500651 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.500626    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/144f56a1-c1f8-4e71-96a1-6a46fc9a42c2-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr\" (UID: \"144f56a1-c1f8-4e71-96a1-6a46fc9a42c2\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr"
Sep 06 19:22:09.500779 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.500666    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl7hr\" (UniqueName: \"kubernetes.io/projected/9fc87590-9575-4698-94ac-ae6804b13a1b-kube-api-access-jl7hr\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k\" (UID: \"9fc87590-9575-4698-94ac-ae6804b13a1b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k"
Sep 06 19:22:09.500779 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.500698    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qkjb\" (UniqueName: \"kubernetes.io/projected/144f56a1-c1f8-4e71-96a1-6a46fc9a42c2-kube-api-access-2qkjb\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr\" (UID: \"144f56a1-c1f8-4e71-96a1-6a46fc9a42c2\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr"
Sep 06 19:22:09.500779 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.500766    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9fc87590-9575-4698-94ac-ae6804b13a1b-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k\" (UID: \"9fc87590-9575-4698-94ac-ae6804b13a1b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k"
Sep 06 19:22:09.500905 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.500805    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/144f56a1-c1f8-4e71-96a1-6a46fc9a42c2-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr\" (UID: \"144f56a1-c1f8-4e71-96a1-6a46fc9a42c2\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr"
Sep 06 19:22:09.500905 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.500839    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9fc87590-9575-4698-94ac-ae6804b13a1b-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k\" (UID: \"9fc87590-9575-4698-94ac-ae6804b13a1b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k"
Sep 06 19:22:09.501110 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.501092    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9fc87590-9575-4698-94ac-ae6804b13a1b-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k\" (UID: \"9fc87590-9575-4698-94ac-ae6804b13a1b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k"
Sep 06 19:22:09.501145 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.501125    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9fc87590-9575-4698-94ac-ae6804b13a1b-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k\" (UID: \"9fc87590-9575-4698-94ac-ae6804b13a1b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k"
Sep 06 19:22:09.507939 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.507923    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9"
Sep 06 19:22:09.508948 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.508923    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl7hr\" (UniqueName: \"kubernetes.io/projected/9fc87590-9575-4698-94ac-ae6804b13a1b-kube-api-access-jl7hr\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k\" (UID: \"9fc87590-9575-4698-94ac-ae6804b13a1b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k"
Sep 06 19:22:09.601972 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.601938    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/144f56a1-c1f8-4e71-96a1-6a46fc9a42c2-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr\" (UID: \"144f56a1-c1f8-4e71-96a1-6a46fc9a42c2\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr"
Sep 06 19:22:09.602105 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.602004    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/144f56a1-c1f8-4e71-96a1-6a46fc9a42c2-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr\" (UID: \"144f56a1-c1f8-4e71-96a1-6a46fc9a42c2\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr"
Sep 06 19:22:09.602105 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.602050    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qkjb\" (UniqueName: \"kubernetes.io/projected/144f56a1-c1f8-4e71-96a1-6a46fc9a42c2-kube-api-access-2qkjb\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr\" (UID: \"144f56a1-c1f8-4e71-96a1-6a46fc9a42c2\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr"
Sep 06 19:22:09.602316 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.602299    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/144f56a1-c1f8-4e71-96a1-6a46fc9a42c2-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr\" (UID: \"144f56a1-c1f8-4e71-96a1-6a46fc9a42c2\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr"
Sep 06 19:22:09.602391 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.602371    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/144f56a1-c1f8-4e71-96a1-6a46fc9a42c2-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr\" (UID: \"144f56a1-c1f8-4e71-96a1-6a46fc9a42c2\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr"
Sep 06 19:22:09.606859 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.606840    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k"
Sep 06 19:22:09.610146 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.610118    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qkjb\" (UniqueName: \"kubernetes.io/projected/144f56a1-c1f8-4e71-96a1-6a46fc9a42c2-kube-api-access-2qkjb\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr\" (UID: \"144f56a1-c1f8-4e71-96a1-6a46fc9a42c2\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr"
Sep 06 19:22:09.683179 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.683153    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf"]
Sep 06 19:22:09.684357 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:22:09.684333    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6cea4f_e1e5_4aa6_bc12_06bbb7af2cba.slice/crio-7d66fe96ffe7fb27aaebeaaf22adb12b1e04a35090d97711478df2cfd39a0323 WatchSource:0}: Error finding container 7d66fe96ffe7fb27aaebeaaf22adb12b1e04a35090d97711478df2cfd39a0323: Status 404 returned error can't find the container with id 7d66fe96ffe7fb27aaebeaaf22adb12b1e04a35090d97711478df2cfd39a0323
Sep 06 19:22:09.709692 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.709672    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr"
Sep 06 19:22:09.778561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.778529    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9"]
Sep 06 19:22:09.779706 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:22:09.779680    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d3f50a_003b_4018_915a_ed1f71f1c647.slice/crio-819a03120bfc7fc90b44681b2087df15e337da16478993f1904b85e99b5c1ba8 WatchSource:0}: Error finding container 819a03120bfc7fc90b44681b2087df15e337da16478993f1904b85e99b5c1ba8: Status 404 returned error can't find the container with id 819a03120bfc7fc90b44681b2087df15e337da16478993f1904b85e99b5c1ba8
Sep 06 19:22:09.877989 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.877955    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9" event={"ID":"c6d3f50a-003b-4018-915a-ed1f71f1c647","Type":"ContainerStarted","Data":"819a03120bfc7fc90b44681b2087df15e337da16478993f1904b85e99b5c1ba8"}
Sep 06 19:22:09.878398 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.878369    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k"]
Sep 06 19:22:09.879791 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.879763    2566 generic.go:356] "Generic (PLEG): container finished" podID="6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba" containerID="afa3b8e49f8bf8d9810a3454e9ec36975151a694b33a7701e62b552779b5ccb4" exitCode=0
Sep 06 19:22:09.879888 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.879815    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf" event={"ID":"6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba","Type":"ContainerDied","Data":"afa3b8e49f8bf8d9810a3454e9ec36975151a694b33a7701e62b552779b5ccb4"}
Sep 06 19:22:09.879888 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.879847    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf" event={"ID":"6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba","Type":"ContainerStarted","Data":"7d66fe96ffe7fb27aaebeaaf22adb12b1e04a35090d97711478df2cfd39a0323"}
Sep 06 19:22:09.880312 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:22:09.880287    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc87590_9575_4698_94ac_ae6804b13a1b.slice/crio-61da2e12a9eda8d56729e01ac86864698a2e2c9443904e23062baa785d0c1bb5 WatchSource:0}: Error finding container 61da2e12a9eda8d56729e01ac86864698a2e2c9443904e23062baa785d0c1bb5: Status 404 returned error can't find the container with id 61da2e12a9eda8d56729e01ac86864698a2e2c9443904e23062baa785d0c1bb5
Sep 06 19:22:09.880998 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.880966    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 19:22:09.984397 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:09.984306    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr"]
Sep 06 19:22:09.997990 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:22:09.997961    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144f56a1_c1f8_4e71_96a1_6a46fc9a42c2.slice/crio-1377e7a919f1c2984f36a11d7f20896ae456ed5e9187a16e42e1f548ed11b839 WatchSource:0}: Error finding container 1377e7a919f1c2984f36a11d7f20896ae456ed5e9187a16e42e1f548ed11b839: Status 404 returned error can't find the container with id 1377e7a919f1c2984f36a11d7f20896ae456ed5e9187a16e42e1f548ed11b839
Sep 06 19:22:10.886172 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:10.886146    2566 generic.go:356] "Generic (PLEG): container finished" podID="c6d3f50a-003b-4018-915a-ed1f71f1c647" containerID="a47b4c68149fbcd0f18a4996c36c8fc4d8348b23c67e85a03483af6ab604907d" exitCode=0
Sep 06 19:22:10.886502 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:10.886228    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9" event={"ID":"c6d3f50a-003b-4018-915a-ed1f71f1c647","Type":"ContainerDied","Data":"a47b4c68149fbcd0f18a4996c36c8fc4d8348b23c67e85a03483af6ab604907d"}
Sep 06 19:22:10.887888 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:10.887866    2566 generic.go:356] "Generic (PLEG): container finished" podID="144f56a1-c1f8-4e71-96a1-6a46fc9a42c2" containerID="3b534bc86d45d0a70022b9596630f7fd3381d6486d0c9e24dd1f47ab8a4b7898" exitCode=0
Sep 06 19:22:10.887968 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:10.887920    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr" event={"ID":"144f56a1-c1f8-4e71-96a1-6a46fc9a42c2","Type":"ContainerDied","Data":"3b534bc86d45d0a70022b9596630f7fd3381d6486d0c9e24dd1f47ab8a4b7898"}
Sep 06 19:22:10.888031 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:10.887975    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr" event={"ID":"144f56a1-c1f8-4e71-96a1-6a46fc9a42c2","Type":"ContainerStarted","Data":"1377e7a919f1c2984f36a11d7f20896ae456ed5e9187a16e42e1f548ed11b839"}
Sep 06 19:22:10.890041 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:10.890017    2566 generic.go:356] "Generic (PLEG): container finished" podID="6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba" containerID="16fb972f8a3fe42fc88118d0f2a15efccbb17e1cf66dfcf9a824f1ffb06c0ae4" exitCode=0
Sep 06 19:22:10.890136 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:10.890119    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf" event={"ID":"6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba","Type":"ContainerDied","Data":"16fb972f8a3fe42fc88118d0f2a15efccbb17e1cf66dfcf9a824f1ffb06c0ae4"}
Sep 06 19:22:10.891796 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:10.891774    2566 generic.go:356] "Generic (PLEG): container finished" podID="9fc87590-9575-4698-94ac-ae6804b13a1b" containerID="c25eee8acb51234276aecb1d303686b4681fc53c7b8f2b5d3eeab1fa907c08c7" exitCode=0
Sep 06 19:22:10.891891 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:10.891816    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k" event={"ID":"9fc87590-9575-4698-94ac-ae6804b13a1b","Type":"ContainerDied","Data":"c25eee8acb51234276aecb1d303686b4681fc53c7b8f2b5d3eeab1fa907c08c7"}
Sep 06 19:22:10.891891 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:10.891841    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k" event={"ID":"9fc87590-9575-4698-94ac-ae6804b13a1b","Type":"ContainerStarted","Data":"61da2e12a9eda8d56729e01ac86864698a2e2c9443904e23062baa785d0c1bb5"}
Sep 06 19:22:11.899226 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:11.899199    2566 generic.go:356] "Generic (PLEG): container finished" podID="6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba" containerID="b430f3623155775f5e6e137f532c8f49ddebfc95af6cbb34224bd59bf1009053" exitCode=0
Sep 06 19:22:11.899551 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:11.899279    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf" event={"ID":"6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba","Type":"ContainerDied","Data":"b430f3623155775f5e6e137f532c8f49ddebfc95af6cbb34224bd59bf1009053"}
Sep 06 19:22:12.909135 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:12.909103    2566 generic.go:356] "Generic (PLEG): container finished" podID="9fc87590-9575-4698-94ac-ae6804b13a1b" containerID="9e01f99648775429c6e7c9e87a3ef706dfb81d751740a4fa597d1c3d99d40061" exitCode=0
Sep 06 19:22:12.909544 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:12.909192    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k" event={"ID":"9fc87590-9575-4698-94ac-ae6804b13a1b","Type":"ContainerDied","Data":"9e01f99648775429c6e7c9e87a3ef706dfb81d751740a4fa597d1c3d99d40061"}
Sep 06 19:22:12.911094 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:12.911075    2566 generic.go:356] "Generic (PLEG): container finished" podID="c6d3f50a-003b-4018-915a-ed1f71f1c647" containerID="9dcd3b5da7dbc145a54db4600071ee5512f60eceaacc65d466c0c1d0192027a2" exitCode=0
Sep 06 19:22:12.911180 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:12.911152    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9" event={"ID":"c6d3f50a-003b-4018-915a-ed1f71f1c647","Type":"ContainerDied","Data":"9dcd3b5da7dbc145a54db4600071ee5512f60eceaacc65d466c0c1d0192027a2"}
Sep 06 19:22:12.913135 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:12.913117    2566 generic.go:356] "Generic (PLEG): container finished" podID="144f56a1-c1f8-4e71-96a1-6a46fc9a42c2" containerID="1b5d3b72393866aefc1747099c0a47deefe74e041e74b2f7f680f3e61beb7ae8" exitCode=0
Sep 06 19:22:12.913243 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:12.913203    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr" event={"ID":"144f56a1-c1f8-4e71-96a1-6a46fc9a42c2","Type":"ContainerDied","Data":"1b5d3b72393866aefc1747099c0a47deefe74e041e74b2f7f680f3e61beb7ae8"}
Sep 06 19:22:13.064102 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:13.064076    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf"
Sep 06 19:22:13.133686 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:13.133661    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba-bundle\") pod \"6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba\" (UID: \"6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba\") "
Sep 06 19:22:13.133791 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:13.133754    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba-util\") pod \"6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba\" (UID: \"6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba\") "
Sep 06 19:22:13.133791 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:13.133776    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cfj6\" (UniqueName: \"kubernetes.io/projected/6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba-kube-api-access-8cfj6\") pod \"6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba\" (UID: \"6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba\") "
Sep 06 19:22:13.134179 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:13.134155    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba-bundle" (OuterVolumeSpecName: "bundle") pod "6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba" (UID: "6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:22:13.135964 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:13.135942    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba-kube-api-access-8cfj6" (OuterVolumeSpecName: "kube-api-access-8cfj6") pod "6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba" (UID: "6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba"). InnerVolumeSpecName "kube-api-access-8cfj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:22:13.139298 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:13.139278    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba-util" (OuterVolumeSpecName: "util") pod "6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba" (UID: "6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:22:13.234757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:13.234735    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba-bundle\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:22:13.234757 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:13.234758    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba-util\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:22:13.234880 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:13.234767    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cfj6\" (UniqueName: \"kubernetes.io/projected/6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba-kube-api-access-8cfj6\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:22:13.920481 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:13.920446    2566 generic.go:356] "Generic (PLEG): container finished" podID="144f56a1-c1f8-4e71-96a1-6a46fc9a42c2" containerID="9489fd23970be0062821b5dca850fce072d4b8e56d0936cf31bb019d3f0a714d" exitCode=0
Sep 06 19:22:13.920935 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:13.920531    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr" event={"ID":"144f56a1-c1f8-4e71-96a1-6a46fc9a42c2","Type":"ContainerDied","Data":"9489fd23970be0062821b5dca850fce072d4b8e56d0936cf31bb019d3f0a714d"}
Sep 06 19:22:13.922549 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:13.922534    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf"
Sep 06 19:22:13.922632 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:13.922558    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb7whf" event={"ID":"6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba","Type":"ContainerDied","Data":"7d66fe96ffe7fb27aaebeaaf22adb12b1e04a35090d97711478df2cfd39a0323"}
Sep 06 19:22:13.922632 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:13.922588    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d66fe96ffe7fb27aaebeaaf22adb12b1e04a35090d97711478df2cfd39a0323"
Sep 06 19:22:13.924791 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:13.924764    2566 generic.go:356] "Generic (PLEG): container finished" podID="9fc87590-9575-4698-94ac-ae6804b13a1b" containerID="a06a9043634ce41fb23c87415149e25df0a50a8d6af8265061176df877cf395f" exitCode=0
Sep 06 19:22:13.924892 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:13.924879    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k" event={"ID":"9fc87590-9575-4698-94ac-ae6804b13a1b","Type":"ContainerDied","Data":"a06a9043634ce41fb23c87415149e25df0a50a8d6af8265061176df877cf395f"}
Sep 06 19:22:13.926898 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:13.926877    2566 generic.go:356] "Generic (PLEG): container finished" podID="c6d3f50a-003b-4018-915a-ed1f71f1c647" containerID="61714754b25e3e8026300f3d5fdd610d82e2b6b879d0c3f86b56bd9c93225b81" exitCode=0
Sep 06 19:22:13.926980 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:13.926902    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9" event={"ID":"c6d3f50a-003b-4018-915a-ed1f71f1c647","Type":"ContainerDied","Data":"61714754b25e3e8026300f3d5fdd610d82e2b6b879d0c3f86b56bd9c93225b81"}
Sep 06 19:22:15.069452 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.069429    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k"
Sep 06 19:22:15.123752 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.123731    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9"
Sep 06 19:22:15.127161 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.127141    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr"
Sep 06 19:22:15.151821 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.151796    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jl7hr\" (UniqueName: \"kubernetes.io/projected/9fc87590-9575-4698-94ac-ae6804b13a1b-kube-api-access-jl7hr\") pod \"9fc87590-9575-4698-94ac-ae6804b13a1b\" (UID: \"9fc87590-9575-4698-94ac-ae6804b13a1b\") "
Sep 06 19:22:15.151964 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.151838    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9fc87590-9575-4698-94ac-ae6804b13a1b-util\") pod \"9fc87590-9575-4698-94ac-ae6804b13a1b\" (UID: \"9fc87590-9575-4698-94ac-ae6804b13a1b\") "
Sep 06 19:22:15.151964 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.151877    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9fc87590-9575-4698-94ac-ae6804b13a1b-bundle\") pod \"9fc87590-9575-4698-94ac-ae6804b13a1b\" (UID: \"9fc87590-9575-4698-94ac-ae6804b13a1b\") "
Sep 06 19:22:15.152693 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.152597    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc87590-9575-4698-94ac-ae6804b13a1b-bundle" (OuterVolumeSpecName: "bundle") pod "9fc87590-9575-4698-94ac-ae6804b13a1b" (UID: "9fc87590-9575-4698-94ac-ae6804b13a1b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:22:15.154091 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.154066    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc87590-9575-4698-94ac-ae6804b13a1b-kube-api-access-jl7hr" (OuterVolumeSpecName: "kube-api-access-jl7hr") pod "9fc87590-9575-4698-94ac-ae6804b13a1b" (UID: "9fc87590-9575-4698-94ac-ae6804b13a1b"). InnerVolumeSpecName "kube-api-access-jl7hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:22:15.253081 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.253055    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6d3f50a-003b-4018-915a-ed1f71f1c647-util\") pod \"c6d3f50a-003b-4018-915a-ed1f71f1c647\" (UID: \"c6d3f50a-003b-4018-915a-ed1f71f1c647\") "
Sep 06 19:22:15.253217 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.253084    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qkjb\" (UniqueName: \"kubernetes.io/projected/144f56a1-c1f8-4e71-96a1-6a46fc9a42c2-kube-api-access-2qkjb\") pod \"144f56a1-c1f8-4e71-96a1-6a46fc9a42c2\" (UID: \"144f56a1-c1f8-4e71-96a1-6a46fc9a42c2\") "
Sep 06 19:22:15.253217 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.253130    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcg82\" (UniqueName: \"kubernetes.io/projected/c6d3f50a-003b-4018-915a-ed1f71f1c647-kube-api-access-hcg82\") pod \"c6d3f50a-003b-4018-915a-ed1f71f1c647\" (UID: \"c6d3f50a-003b-4018-915a-ed1f71f1c647\") "
Sep 06 19:22:15.253217 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.253152    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/144f56a1-c1f8-4e71-96a1-6a46fc9a42c2-bundle\") pod \"144f56a1-c1f8-4e71-96a1-6a46fc9a42c2\" (UID: \"144f56a1-c1f8-4e71-96a1-6a46fc9a42c2\") "
Sep 06 19:22:15.253217 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.253185    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/144f56a1-c1f8-4e71-96a1-6a46fc9a42c2-util\") pod \"144f56a1-c1f8-4e71-96a1-6a46fc9a42c2\" (UID: \"144f56a1-c1f8-4e71-96a1-6a46fc9a42c2\") "
Sep 06 19:22:15.253217 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.253202    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6d3f50a-003b-4018-915a-ed1f71f1c647-bundle\") pod \"c6d3f50a-003b-4018-915a-ed1f71f1c647\" (UID: \"c6d3f50a-003b-4018-915a-ed1f71f1c647\") "
Sep 06 19:22:15.253486 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.253454    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl7hr\" (UniqueName: \"kubernetes.io/projected/9fc87590-9575-4698-94ac-ae6804b13a1b-kube-api-access-jl7hr\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:22:15.253486 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.253473    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9fc87590-9575-4698-94ac-ae6804b13a1b-bundle\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:22:15.253746 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.253720    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144f56a1-c1f8-4e71-96a1-6a46fc9a42c2-bundle" (OuterVolumeSpecName: "bundle") pod "144f56a1-c1f8-4e71-96a1-6a46fc9a42c2" (UID: "144f56a1-c1f8-4e71-96a1-6a46fc9a42c2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:22:15.253802 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.253786    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d3f50a-003b-4018-915a-ed1f71f1c647-bundle" (OuterVolumeSpecName: "bundle") pod "c6d3f50a-003b-4018-915a-ed1f71f1c647" (UID: "c6d3f50a-003b-4018-915a-ed1f71f1c647"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:22:15.255259 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.255238    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144f56a1-c1f8-4e71-96a1-6a46fc9a42c2-kube-api-access-2qkjb" (OuterVolumeSpecName: "kube-api-access-2qkjb") pod "144f56a1-c1f8-4e71-96a1-6a46fc9a42c2" (UID: "144f56a1-c1f8-4e71-96a1-6a46fc9a42c2"). InnerVolumeSpecName "kube-api-access-2qkjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:22:15.255315 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.255271    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d3f50a-003b-4018-915a-ed1f71f1c647-kube-api-access-hcg82" (OuterVolumeSpecName: "kube-api-access-hcg82") pod "c6d3f50a-003b-4018-915a-ed1f71f1c647" (UID: "c6d3f50a-003b-4018-915a-ed1f71f1c647"). InnerVolumeSpecName "kube-api-access-hcg82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:22:15.353829 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.353807    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/144f56a1-c1f8-4e71-96a1-6a46fc9a42c2-bundle\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:22:15.353829 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.353826    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6d3f50a-003b-4018-915a-ed1f71f1c647-bundle\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:22:15.353936 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.353835    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qkjb\" (UniqueName: \"kubernetes.io/projected/144f56a1-c1f8-4e71-96a1-6a46fc9a42c2-kube-api-access-2qkjb\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:22:15.353936 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.353846    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcg82\" (UniqueName: \"kubernetes.io/projected/c6d3f50a-003b-4018-915a-ed1f71f1c647-kube-api-access-hcg82\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:22:15.453233 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.453209    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc87590-9575-4698-94ac-ae6804b13a1b-util" (OuterVolumeSpecName: "util") pod "9fc87590-9575-4698-94ac-ae6804b13a1b" (UID: "9fc87590-9575-4698-94ac-ae6804b13a1b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:22:15.454593 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.454575    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9fc87590-9575-4698-94ac-ae6804b13a1b-util\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:22:15.456586 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.456565    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d3f50a-003b-4018-915a-ed1f71f1c647-util" (OuterVolumeSpecName: "util") pod "c6d3f50a-003b-4018-915a-ed1f71f1c647" (UID: "c6d3f50a-003b-4018-915a-ed1f71f1c647"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:22:15.456889 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.456870    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144f56a1-c1f8-4e71-96a1-6a46fc9a42c2-util" (OuterVolumeSpecName: "util") pod "144f56a1-c1f8-4e71-96a1-6a46fc9a42c2" (UID: "144f56a1-c1f8-4e71-96a1-6a46fc9a42c2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:22:15.555205 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.555160    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6d3f50a-003b-4018-915a-ed1f71f1c647-util\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:22:15.555205 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.555178    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/144f56a1-c1f8-4e71-96a1-6a46fc9a42c2-util\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:22:15.940062 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.939972    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr"
Sep 06 19:22:15.940062 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.939994    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0sb8zr" event={"ID":"144f56a1-c1f8-4e71-96a1-6a46fc9a42c2","Type":"ContainerDied","Data":"1377e7a919f1c2984f36a11d7f20896ae456ed5e9187a16e42e1f548ed11b839"}
Sep 06 19:22:15.940062 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.940042    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1377e7a919f1c2984f36a11d7f20896ae456ed5e9187a16e42e1f548ed11b839"
Sep 06 19:22:15.942100 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.942081    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k"
Sep 06 19:22:15.942227 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.942113    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534ht4k" event={"ID":"9fc87590-9575-4698-94ac-ae6804b13a1b","Type":"ContainerDied","Data":"61da2e12a9eda8d56729e01ac86864698a2e2c9443904e23062baa785d0c1bb5"}
Sep 06 19:22:15.942227 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.942153    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61da2e12a9eda8d56729e01ac86864698a2e2c9443904e23062baa785d0c1bb5"
Sep 06 19:22:15.944174 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.944150    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9" event={"ID":"c6d3f50a-003b-4018-915a-ed1f71f1c647","Type":"ContainerDied","Data":"819a03120bfc7fc90b44681b2087df15e337da16478993f1904b85e99b5c1ba8"}
Sep 06 19:22:15.944278 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.944181    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="819a03120bfc7fc90b44681b2087df15e337da16478993f1904b85e99b5c1ba8"
Sep 06 19:22:15.944278 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:15.944187    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhh7m9"
Sep 06 19:22:31.624389 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624346    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-xkhvn"]
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624722    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fc87590-9575-4698-94ac-ae6804b13a1b" containerName="pull"
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624735    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fc87590-9575-4698-94ac-ae6804b13a1b" containerName="pull"
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624746    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fc87590-9575-4698-94ac-ae6804b13a1b" containerName="util"
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624752    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fc87590-9575-4698-94ac-ae6804b13a1b" containerName="util"
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624758    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6d3f50a-003b-4018-915a-ed1f71f1c647" containerName="pull"
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624765    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6d3f50a-003b-4018-915a-ed1f71f1c647" containerName="pull"
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624780    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="144f56a1-c1f8-4e71-96a1-6a46fc9a42c2" containerName="extract"
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624785    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="144f56a1-c1f8-4e71-96a1-6a46fc9a42c2" containerName="extract"
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624796    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba" containerName="util"
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624801    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba" containerName="util"
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624808    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="144f56a1-c1f8-4e71-96a1-6a46fc9a42c2" containerName="util"
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624812    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="144f56a1-c1f8-4e71-96a1-6a46fc9a42c2" containerName="util"
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624818    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba" containerName="extract"
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624823    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba" containerName="extract"
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624833    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6d3f50a-003b-4018-915a-ed1f71f1c647" containerName="util"
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624837    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6d3f50a-003b-4018-915a-ed1f71f1c647" containerName="util"
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624846    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6d3f50a-003b-4018-915a-ed1f71f1c647" containerName="extract"
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624854    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6d3f50a-003b-4018-915a-ed1f71f1c647" containerName="extract"
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624862    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba" containerName="pull"
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624867    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba" containerName="pull"
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624872    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="144f56a1-c1f8-4e71-96a1-6a46fc9a42c2" containerName="pull"
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624877    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="144f56a1-c1f8-4e71-96a1-6a46fc9a42c2" containerName="pull"
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624881    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fc87590-9575-4698-94ac-ae6804b13a1b" containerName="extract"
Sep 06 19:22:31.624887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624886    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fc87590-9575-4698-94ac-ae6804b13a1b" containerName="extract"
Sep 06 19:22:31.626020 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624939    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6d3f50a-003b-4018-915a-ed1f71f1c647" containerName="extract"
Sep 06 19:22:31.626020 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624948    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b6cea4f-e1e5-4aa6-bc12-06bbb7af2cba" containerName="extract"
Sep 06 19:22:31.626020 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624956    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fc87590-9575-4698-94ac-ae6804b13a1b" containerName="extract"
Sep 06 19:22:31.626020 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.624962    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="144f56a1-c1f8-4e71-96a1-6a46fc9a42c2" containerName="extract"
Sep 06 19:22:31.627815 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.627797    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-xkhvn"
Sep 06 19:22:31.630319 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.630292    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 06 19:22:31.630432 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.630319    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 06 19:22:31.631198 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.631183    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 06 19:22:31.631261 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.631198    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 06 19:22:31.631314 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.631192    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-px5nx\""
Sep 06 19:22:31.635685 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.635666    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-xkhvn"]
Sep 06 19:22:31.689119 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.689092    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6cb66892-087e-4585-af7d-3a56bf05144f-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-xkhvn\" (UID: \"6cb66892-087e-4585-af7d-3a56bf05144f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-xkhvn"
Sep 06 19:22:31.689357 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.689128    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cb66892-087e-4585-af7d-3a56bf05144f-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-xkhvn\" (UID: \"6cb66892-087e-4585-af7d-3a56bf05144f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-xkhvn"
Sep 06 19:22:31.689357 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.689152    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxhct\" (UniqueName: \"kubernetes.io/projected/6cb66892-087e-4585-af7d-3a56bf05144f-kube-api-access-jxhct\") pod \"kuadrant-console-plugin-598f86468b-xkhvn\" (UID: \"6cb66892-087e-4585-af7d-3a56bf05144f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-xkhvn"
Sep 06 19:22:31.790447 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.790418    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6cb66892-087e-4585-af7d-3a56bf05144f-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-xkhvn\" (UID: \"6cb66892-087e-4585-af7d-3a56bf05144f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-xkhvn"
Sep 06 19:22:31.790447 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.790452    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cb66892-087e-4585-af7d-3a56bf05144f-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-xkhvn\" (UID: \"6cb66892-087e-4585-af7d-3a56bf05144f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-xkhvn"
Sep 06 19:22:31.790753 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.790476    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxhct\" (UniqueName: \"kubernetes.io/projected/6cb66892-087e-4585-af7d-3a56bf05144f-kube-api-access-jxhct\") pod \"kuadrant-console-plugin-598f86468b-xkhvn\" (UID: \"6cb66892-087e-4585-af7d-3a56bf05144f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-xkhvn"
Sep 06 19:22:31.790753 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:22:31.790589    2566 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 06 19:22:31.790753 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:22:31.790691    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb66892-087e-4585-af7d-3a56bf05144f-plugin-serving-cert podName:6cb66892-087e-4585-af7d-3a56bf05144f nodeName:}" failed. No retries permitted until 2026-09-06 19:22:32.290674478 +0000 UTC m=+686.471856262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/6cb66892-087e-4585-af7d-3a56bf05144f-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-xkhvn" (UID: "6cb66892-087e-4585-af7d-3a56bf05144f") : secret "plugin-serving-cert" not found
Sep 06 19:22:31.791076 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.791058    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6cb66892-087e-4585-af7d-3a56bf05144f-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-xkhvn\" (UID: \"6cb66892-087e-4585-af7d-3a56bf05144f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-xkhvn"
Sep 06 19:22:31.798732 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:31.798713    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxhct\" (UniqueName: \"kubernetes.io/projected/6cb66892-087e-4585-af7d-3a56bf05144f-kube-api-access-jxhct\") pod \"kuadrant-console-plugin-598f86468b-xkhvn\" (UID: \"6cb66892-087e-4585-af7d-3a56bf05144f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-xkhvn"
Sep 06 19:22:32.296295 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:32.296256    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cb66892-087e-4585-af7d-3a56bf05144f-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-xkhvn\" (UID: \"6cb66892-087e-4585-af7d-3a56bf05144f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-xkhvn"
Sep 06 19:22:32.298812 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:32.298782    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cb66892-087e-4585-af7d-3a56bf05144f-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-xkhvn\" (UID: \"6cb66892-087e-4585-af7d-3a56bf05144f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-xkhvn"
Sep 06 19:22:32.538577 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:32.538547    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-xkhvn"
Sep 06 19:22:32.806670 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:32.806583    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-xkhvn"]
Sep 06 19:22:32.809085 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:22:32.809058    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cb66892_087e_4585_af7d_3a56bf05144f.slice/crio-b3a336e6152a155f5a86be1d22a54cad7700e99b69bb133b7cb5fb71efed848b WatchSource:0}: Error finding container b3a336e6152a155f5a86be1d22a54cad7700e99b69bb133b7cb5fb71efed848b: Status 404 returned error can't find the container with id b3a336e6152a155f5a86be1d22a54cad7700e99b69bb133b7cb5fb71efed848b
Sep 06 19:22:33.040849 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:33.040813    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-xkhvn" event={"ID":"6cb66892-087e-4585-af7d-3a56bf05144f","Type":"ContainerStarted","Data":"b3a336e6152a155f5a86be1d22a54cad7700e99b69bb133b7cb5fb71efed848b"}
Sep 06 19:22:39.081256 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:39.081220    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-xkhvn" event={"ID":"6cb66892-087e-4585-af7d-3a56bf05144f","Type":"ContainerStarted","Data":"f542f5b3cc8b1d245a4c74e8b4f65e5cd068d0bee4c2bd2f887c2a9a0f83fbf2"}
Sep 06 19:22:39.097373 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:22:39.097323    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-xkhvn" podStartSLOduration=2.216637929 podStartE2EDuration="8.097309871s" totalImagesPullingTime="5.880671942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:22:39.095529513 +0000 UTC m=+693.276711321" watchObservedRunningTime="2026-09-06 19:22:39.097309871 +0000 UTC m=+693.278491677"
Sep 06 19:23:25.446718 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:25.446688    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:23:25.467668 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:25.467637    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:23:25.467825 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:25.467703    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"
Sep 06 19:23:25.470259 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:25.470240    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 06 19:23:25.473721 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:25.473700    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:23:25.640030 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:25.639989    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af6ca84e-0ad3-4242-9918-5f0a4444245e-config-file\") pod \"limitador-limitador-85d9db4d5d-l5dmn\" (UID: \"af6ca84e-0ad3-4242-9918-5f0a4444245e\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"
Sep 06 19:23:25.640220 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:25.640134    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvvxd\" (UniqueName: \"kubernetes.io/projected/af6ca84e-0ad3-4242-9918-5f0a4444245e-kube-api-access-fvvxd\") pod \"limitador-limitador-85d9db4d5d-l5dmn\" (UID: \"af6ca84e-0ad3-4242-9918-5f0a4444245e\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"
Sep 06 19:23:25.741531 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:25.741498    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvvxd\" (UniqueName: \"kubernetes.io/projected/af6ca84e-0ad3-4242-9918-5f0a4444245e-kube-api-access-fvvxd\") pod \"limitador-limitador-85d9db4d5d-l5dmn\" (UID: \"af6ca84e-0ad3-4242-9918-5f0a4444245e\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"
Sep 06 19:23:25.741715 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:25.741539    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af6ca84e-0ad3-4242-9918-5f0a4444245e-config-file\") pod \"limitador-limitador-85d9db4d5d-l5dmn\" (UID: \"af6ca84e-0ad3-4242-9918-5f0a4444245e\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"
Sep 06 19:23:25.742140 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:25.742121    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af6ca84e-0ad3-4242-9918-5f0a4444245e-config-file\") pod \"limitador-limitador-85d9db4d5d-l5dmn\" (UID: \"af6ca84e-0ad3-4242-9918-5f0a4444245e\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"
Sep 06 19:23:25.749208 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:25.749186    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvvxd\" (UniqueName: \"kubernetes.io/projected/af6ca84e-0ad3-4242-9918-5f0a4444245e-kube-api-access-fvvxd\") pod \"limitador-limitador-85d9db4d5d-l5dmn\" (UID: \"af6ca84e-0ad3-4242-9918-5f0a4444245e\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"
Sep 06 19:23:25.778529 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:25.778502    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"
Sep 06 19:23:25.942035 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:25.942007    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-xj64h"]
Sep 06 19:23:25.945844 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:25.945829    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-xj64h"
Sep 06 19:23:25.948098 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:25.948082    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8fwdc\""
Sep 06 19:23:25.955051 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:25.955031    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-xj64h"]
Sep 06 19:23:26.044628 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:26.044539    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47txt\" (UniqueName: \"kubernetes.io/projected/75b63ddb-580a-4e62-9b0d-295d315a3494-kube-api-access-47txt\") pod \"authorino-b54df94b4-xj64h\" (UID: \"75b63ddb-580a-4e62-9b0d-295d315a3494\") " pod="kuadrant-system/authorino-b54df94b4-xj64h"
Sep 06 19:23:26.063395 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:26.063378    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:23:26.065026 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:23:26.065003    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf6ca84e_0ad3_4242_9918_5f0a4444245e.slice/crio-f5f959550d594f8af4fc0d8dea37597b2a131f9a0cb2a41d247dadb411d8f65b WatchSource:0}: Error finding container f5f959550d594f8af4fc0d8dea37597b2a131f9a0cb2a41d247dadb411d8f65b: Status 404 returned error can't find the container with id f5f959550d594f8af4fc0d8dea37597b2a131f9a0cb2a41d247dadb411d8f65b
Sep 06 19:23:26.144944 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:26.144920    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47txt\" (UniqueName: \"kubernetes.io/projected/75b63ddb-580a-4e62-9b0d-295d315a3494-kube-api-access-47txt\") pod \"authorino-b54df94b4-xj64h\" (UID: \"75b63ddb-580a-4e62-9b0d-295d315a3494\") " pod="kuadrant-system/authorino-b54df94b4-xj64h"
Sep 06 19:23:26.159229 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:26.159203    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47txt\" (UniqueName: \"kubernetes.io/projected/75b63ddb-580a-4e62-9b0d-295d315a3494-kube-api-access-47txt\") pod \"authorino-b54df94b4-xj64h\" (UID: \"75b63ddb-580a-4e62-9b0d-295d315a3494\") " pod="kuadrant-system/authorino-b54df94b4-xj64h"
Sep 06 19:23:26.256092 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:26.256068    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-xj64h"
Sep 06 19:23:26.324284 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:26.324257    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn" event={"ID":"af6ca84e-0ad3-4242-9918-5f0a4444245e","Type":"ContainerStarted","Data":"f5f959550d594f8af4fc0d8dea37597b2a131f9a0cb2a41d247dadb411d8f65b"}
Sep 06 19:23:26.524402 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:26.524379    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-xj64h"]
Sep 06 19:23:26.525952 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:23:26.525921    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75b63ddb_580a_4e62_9b0d_295d315a3494.slice/crio-abeb00a091334dad94c34ac211f2a4a2510a06686af849dd649d0d6822e1e175 WatchSource:0}: Error finding container abeb00a091334dad94c34ac211f2a4a2510a06686af849dd649d0d6822e1e175: Status 404 returned error can't find the container with id abeb00a091334dad94c34ac211f2a4a2510a06686af849dd649d0d6822e1e175
Sep 06 19:23:27.335692 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:27.335643    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-xj64h" event={"ID":"75b63ddb-580a-4e62-9b0d-295d315a3494","Type":"ContainerStarted","Data":"abeb00a091334dad94c34ac211f2a4a2510a06686af849dd649d0d6822e1e175"}
Sep 06 19:23:29.346616 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:29.346568    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-xj64h" event={"ID":"75b63ddb-580a-4e62-9b0d-295d315a3494","Type":"ContainerStarted","Data":"79da9b87d0fafa5497869e9299103364d1a0c311ec025b5974f2a835d2955b1a"}
Sep 06 19:23:29.348363 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:29.348338    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn" event={"ID":"af6ca84e-0ad3-4242-9918-5f0a4444245e","Type":"ContainerStarted","Data":"3a8543494e763bd8fdd9fb3bb2c8e3435f0c46da0155488d327a292679448c20"}
Sep 06 19:23:29.348480 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:29.348419    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"
Sep 06 19:23:29.360052 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:29.360002    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-xj64h" podStartSLOduration=2.087295759 podStartE2EDuration="4.35998998s" totalImagesPullingTime="2.272694221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:23:29.359472149 +0000 UTC m=+743.540653956" watchObservedRunningTime="2026-09-06 19:23:29.35998998 +0000 UTC m=+743.541171786"
Sep 06 19:23:29.374070 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:29.374028    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn" podStartSLOduration=1.647164061 podStartE2EDuration="4.374013535s" totalImagesPullingTime="2.726849474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:23:29.373448289 +0000 UTC m=+743.554630097" watchObservedRunningTime="2026-09-06 19:23:29.374013535 +0000 UTC m=+743.555195342"
Sep 06 19:23:29.461225 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:29.461188    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-xj64h"]
Sep 06 19:23:31.358041 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:31.358002    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-xj64h" podUID="75b63ddb-580a-4e62-9b0d-295d315a3494" containerName="authorino" containerID="cri-o://79da9b87d0fafa5497869e9299103364d1a0c311ec025b5974f2a835d2955b1a" gracePeriod=30
Sep 06 19:23:31.610489 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:31.610427    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-xj64h"
Sep 06 19:23:31.799283 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:31.799245    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47txt\" (UniqueName: \"kubernetes.io/projected/75b63ddb-580a-4e62-9b0d-295d315a3494-kube-api-access-47txt\") pod \"75b63ddb-580a-4e62-9b0d-295d315a3494\" (UID: \"75b63ddb-580a-4e62-9b0d-295d315a3494\") "
Sep 06 19:23:31.801754 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:31.801723    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b63ddb-580a-4e62-9b0d-295d315a3494-kube-api-access-47txt" (OuterVolumeSpecName: "kube-api-access-47txt") pod "75b63ddb-580a-4e62-9b0d-295d315a3494" (UID: "75b63ddb-580a-4e62-9b0d-295d315a3494"). InnerVolumeSpecName "kube-api-access-47txt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:23:31.900109 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:31.900027    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47txt\" (UniqueName: \"kubernetes.io/projected/75b63ddb-580a-4e62-9b0d-295d315a3494-kube-api-access-47txt\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:23:32.364219 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:32.364185    2566 generic.go:356] "Generic (PLEG): container finished" podID="75b63ddb-580a-4e62-9b0d-295d315a3494" containerID="79da9b87d0fafa5497869e9299103364d1a0c311ec025b5974f2a835d2955b1a" exitCode=0
Sep 06 19:23:32.364670 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:32.364237    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-xj64h"
Sep 06 19:23:32.364670 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:32.364235    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-xj64h" event={"ID":"75b63ddb-580a-4e62-9b0d-295d315a3494","Type":"ContainerDied","Data":"79da9b87d0fafa5497869e9299103364d1a0c311ec025b5974f2a835d2955b1a"}
Sep 06 19:23:32.364670 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:32.364288    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-xj64h" event={"ID":"75b63ddb-580a-4e62-9b0d-295d315a3494","Type":"ContainerDied","Data":"abeb00a091334dad94c34ac211f2a4a2510a06686af849dd649d0d6822e1e175"}
Sep 06 19:23:32.364670 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:32.364315    2566 scope.go:117] "RemoveContainer" containerID="79da9b87d0fafa5497869e9299103364d1a0c311ec025b5974f2a835d2955b1a"
Sep 06 19:23:32.373426 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:32.373405    2566 scope.go:117] "RemoveContainer" containerID="79da9b87d0fafa5497869e9299103364d1a0c311ec025b5974f2a835d2955b1a"
Sep 06 19:23:32.373715 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:23:32.373699    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79da9b87d0fafa5497869e9299103364d1a0c311ec025b5974f2a835d2955b1a\": container with ID starting with 79da9b87d0fafa5497869e9299103364d1a0c311ec025b5974f2a835d2955b1a not found: ID does not exist" containerID="79da9b87d0fafa5497869e9299103364d1a0c311ec025b5974f2a835d2955b1a"
Sep 06 19:23:32.373775 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:32.373725    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79da9b87d0fafa5497869e9299103364d1a0c311ec025b5974f2a835d2955b1a"} err="failed to get container status \"79da9b87d0fafa5497869e9299103364d1a0c311ec025b5974f2a835d2955b1a\": rpc error: code = NotFound desc = could not find container \"79da9b87d0fafa5497869e9299103364d1a0c311ec025b5974f2a835d2955b1a\": container with ID starting with 79da9b87d0fafa5497869e9299103364d1a0c311ec025b5974f2a835d2955b1a not found: ID does not exist"
Sep 06 19:23:32.379390 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:32.379369    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-xj64h"]
Sep 06 19:23:32.381406 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:32.381384    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-xj64h"]
Sep 06 19:23:34.309627 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:34.309570    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75b63ddb-580a-4e62-9b0d-295d315a3494" path="/var/lib/kubelet/pods/75b63ddb-580a-4e62-9b0d-295d315a3494/volumes"
Sep 06 19:23:40.354411 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:23:40.354380    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"
Sep 06 19:24:40.849767 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:24:40.849678    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:25:01.755312 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.755273    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"]
Sep 06 19:25:01.755713 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.755696    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75b63ddb-580a-4e62-9b0d-295d315a3494" containerName="authorino"
Sep 06 19:25:01.755772 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.755718    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="75b63ddb-580a-4e62-9b0d-295d315a3494" containerName="authorino"
Sep 06 19:25:01.755845 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.755832    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="75b63ddb-580a-4e62-9b0d-295d315a3494" containerName="authorino"
Sep 06 19:25:01.758927 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.758904    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:01.761538 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.761510    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 06 19:25:01.762834 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.762813    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 06 19:25:01.762914 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.762816    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-84kjj\""
Sep 06 19:25:01.762914 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.762813    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 06 19:25:01.766423 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.766405    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"]
Sep 06 19:25:01.842792 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.842757    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsr78\" (UniqueName: \"kubernetes.io/projected/39a3ab5d-0bef-434b-83c9-6e48a52bf8e7-kube-api-access-jsr78\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7\" (UID: \"39a3ab5d-0bef-434b-83c9-6e48a52bf8e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:01.842792 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.842797    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39a3ab5d-0bef-434b-83c9-6e48a52bf8e7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7\" (UID: \"39a3ab5d-0bef-434b-83c9-6e48a52bf8e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:01.843013 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.842824    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39a3ab5d-0bef-434b-83c9-6e48a52bf8e7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7\" (UID: \"39a3ab5d-0bef-434b-83c9-6e48a52bf8e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:01.843013 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.842872    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39a3ab5d-0bef-434b-83c9-6e48a52bf8e7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7\" (UID: \"39a3ab5d-0bef-434b-83c9-6e48a52bf8e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:01.843013 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.842927    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39a3ab5d-0bef-434b-83c9-6e48a52bf8e7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7\" (UID: \"39a3ab5d-0bef-434b-83c9-6e48a52bf8e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:01.843013 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.842957    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39a3ab5d-0bef-434b-83c9-6e48a52bf8e7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7\" (UID: \"39a3ab5d-0bef-434b-83c9-6e48a52bf8e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:01.944037 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.944000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39a3ab5d-0bef-434b-83c9-6e48a52bf8e7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7\" (UID: \"39a3ab5d-0bef-434b-83c9-6e48a52bf8e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:01.944222 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.944050    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsr78\" (UniqueName: \"kubernetes.io/projected/39a3ab5d-0bef-434b-83c9-6e48a52bf8e7-kube-api-access-jsr78\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7\" (UID: \"39a3ab5d-0bef-434b-83c9-6e48a52bf8e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:01.944222 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.944083    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39a3ab5d-0bef-434b-83c9-6e48a52bf8e7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7\" (UID: \"39a3ab5d-0bef-434b-83c9-6e48a52bf8e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:01.944222 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.944114    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39a3ab5d-0bef-434b-83c9-6e48a52bf8e7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7\" (UID: \"39a3ab5d-0bef-434b-83c9-6e48a52bf8e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:01.944222 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.944139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39a3ab5d-0bef-434b-83c9-6e48a52bf8e7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7\" (UID: \"39a3ab5d-0bef-434b-83c9-6e48a52bf8e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:01.944222 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.944207    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39a3ab5d-0bef-434b-83c9-6e48a52bf8e7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7\" (UID: \"39a3ab5d-0bef-434b-83c9-6e48a52bf8e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:01.944544 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.944509    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39a3ab5d-0bef-434b-83c9-6e48a52bf8e7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7\" (UID: \"39a3ab5d-0bef-434b-83c9-6e48a52bf8e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:01.944544 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.944536    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39a3ab5d-0bef-434b-83c9-6e48a52bf8e7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7\" (UID: \"39a3ab5d-0bef-434b-83c9-6e48a52bf8e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:01.944747 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.944560    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39a3ab5d-0bef-434b-83c9-6e48a52bf8e7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7\" (UID: \"39a3ab5d-0bef-434b-83c9-6e48a52bf8e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:01.946576 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.946555    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39a3ab5d-0bef-434b-83c9-6e48a52bf8e7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7\" (UID: \"39a3ab5d-0bef-434b-83c9-6e48a52bf8e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:01.946759 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.946743    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39a3ab5d-0bef-434b-83c9-6e48a52bf8e7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7\" (UID: \"39a3ab5d-0bef-434b-83c9-6e48a52bf8e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:01.951407 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:01.951385    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsr78\" (UniqueName: \"kubernetes.io/projected/39a3ab5d-0bef-434b-83c9-6e48a52bf8e7-kube-api-access-jsr78\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7\" (UID: \"39a3ab5d-0bef-434b-83c9-6e48a52bf8e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:02.069695 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:02.069599    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:02.340111 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:02.340088    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"]
Sep 06 19:25:02.341654 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:25:02.341626    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a3ab5d_0bef_434b_83c9_6e48a52bf8e7.slice/crio-5d5166cc1f78a92940781cca54ad2d82b545e2b9477ac4309a1e681d0ac75e58 WatchSource:0}: Error finding container 5d5166cc1f78a92940781cca54ad2d82b545e2b9477ac4309a1e681d0ac75e58: Status 404 returned error can't find the container with id 5d5166cc1f78a92940781cca54ad2d82b545e2b9477ac4309a1e681d0ac75e58
Sep 06 19:25:02.385551 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:02.385523    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:25:02.822649 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:02.822597    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7" event={"ID":"39a3ab5d-0bef-434b-83c9-6e48a52bf8e7","Type":"ContainerStarted","Data":"5d5166cc1f78a92940781cca54ad2d82b545e2b9477ac4309a1e681d0ac75e58"}
Sep 06 19:25:03.829862 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:03.829773    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7" event={"ID":"39a3ab5d-0bef-434b-83c9-6e48a52bf8e7","Type":"ContainerStarted","Data":"d132c176592b9b75f0231dd73e792484d5cc21b9624407e62d2d0fbb96844c13"}
Sep 06 19:25:03.845622 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:03.845554    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7" podStartSLOduration=1.714947248 podStartE2EDuration="2.845535297s" totalImagesPullingTime="1.130588049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:25:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:25:03.844769671 +0000 UTC m=+838.025951502" watchObservedRunningTime="2026-09-06 19:25:03.845535297 +0000 UTC m=+838.026717104"
Sep 06 19:25:11.842522 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:11.842487    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:25:12.070430 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:12.070397    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:12.070648 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:12.070471    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:12.082723 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:12.082704    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:12.888083 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:12.888052    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7"
Sep 06 19:25:17.444969 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:17.444929    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:25:28.851147 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:28.851111    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:25:37.550391 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:37.550354    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:25:53.247262 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:25:53.247233    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:26:06.342888 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:26:06.342858    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnssq_9254e6f1-b73d-49cc-8bc1-513925ccde4f/ovn-acl-logging/0.log"
Sep 06 19:26:06.344359 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:26:06.344332    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnssq_9254e6f1-b73d-49cc-8bc1-513925ccde4f/ovn-acl-logging/0.log"
Sep 06 19:26:07.145694 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:26:07.145655    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:27:07.757135 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:07.757100    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-9998564fb-k9b7t"]
Sep 06 19:27:07.760666 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:07.760643    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9998564fb-k9b7t"
Sep 06 19:27:07.763004 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:07.762983    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8fwdc\""
Sep 06 19:27:07.764023 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:07.764007    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 06 19:27:07.768062 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:07.768037    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9998564fb-k9b7t"]
Sep 06 19:27:07.867805 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:07.867780    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzmzp\" (UniqueName: \"kubernetes.io/projected/fdb6e7f4-28c5-4486-a8f6-22788f72ea97-kube-api-access-kzmzp\") pod \"authorino-9998564fb-k9b7t\" (UID: \"fdb6e7f4-28c5-4486-a8f6-22788f72ea97\") " pod="kuadrant-system/authorino-9998564fb-k9b7t"
Sep 06 19:27:07.867942 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:07.867819    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fdb6e7f4-28c5-4486-a8f6-22788f72ea97-tls-cert\") pod \"authorino-9998564fb-k9b7t\" (UID: \"fdb6e7f4-28c5-4486-a8f6-22788f72ea97\") " pod="kuadrant-system/authorino-9998564fb-k9b7t"
Sep 06 19:27:07.969027 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:07.968987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzmzp\" (UniqueName: \"kubernetes.io/projected/fdb6e7f4-28c5-4486-a8f6-22788f72ea97-kube-api-access-kzmzp\") pod \"authorino-9998564fb-k9b7t\" (UID: \"fdb6e7f4-28c5-4486-a8f6-22788f72ea97\") " pod="kuadrant-system/authorino-9998564fb-k9b7t"
Sep 06 19:27:07.969152 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:07.969048    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fdb6e7f4-28c5-4486-a8f6-22788f72ea97-tls-cert\") pod \"authorino-9998564fb-k9b7t\" (UID: \"fdb6e7f4-28c5-4486-a8f6-22788f72ea97\") " pod="kuadrant-system/authorino-9998564fb-k9b7t"
Sep 06 19:27:07.971540 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:07.971520    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fdb6e7f4-28c5-4486-a8f6-22788f72ea97-tls-cert\") pod \"authorino-9998564fb-k9b7t\" (UID: \"fdb6e7f4-28c5-4486-a8f6-22788f72ea97\") " pod="kuadrant-system/authorino-9998564fb-k9b7t"
Sep 06 19:27:07.977367 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:07.977337    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzmzp\" (UniqueName: \"kubernetes.io/projected/fdb6e7f4-28c5-4486-a8f6-22788f72ea97-kube-api-access-kzmzp\") pod \"authorino-9998564fb-k9b7t\" (UID: \"fdb6e7f4-28c5-4486-a8f6-22788f72ea97\") " pod="kuadrant-system/authorino-9998564fb-k9b7t"
Sep 06 19:27:08.070790 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:08.070735    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9998564fb-k9b7t"
Sep 06 19:27:08.339835 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:08.339812    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9998564fb-k9b7t"]
Sep 06 19:27:08.341720 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:27:08.341695    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfdb6e7f4_28c5_4486_a8f6_22788f72ea97.slice/crio-f68d168b77f2b7efdaf51eb1096afe91e77941d663030d99ac03b35f41a7957d WatchSource:0}: Error finding container f68d168b77f2b7efdaf51eb1096afe91e77941d663030d99ac03b35f41a7957d: Status 404 returned error can't find the container with id f68d168b77f2b7efdaf51eb1096afe91e77941d663030d99ac03b35f41a7957d
Sep 06 19:27:08.482817 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:08.482789    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9998564fb-k9b7t" event={"ID":"fdb6e7f4-28c5-4486-a8f6-22788f72ea97","Type":"ContainerStarted","Data":"f68d168b77f2b7efdaf51eb1096afe91e77941d663030d99ac03b35f41a7957d"}
Sep 06 19:27:09.491107 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:09.491074    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9998564fb-k9b7t" event={"ID":"fdb6e7f4-28c5-4486-a8f6-22788f72ea97","Type":"ContainerStarted","Data":"822d9892b6a8e9c0409a532921c6df8f843a067a3a2192d9da1eeb93cdc26330"}
Sep 06 19:27:09.508323 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:09.508267    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-9998564fb-k9b7t" podStartSLOduration=1.7919187509999999 podStartE2EDuration="2.508247076s" totalImagesPullingTime="716.328325ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:27:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:27:09.50648838 +0000 UTC m=+963.687670186" watchObservedRunningTime="2026-09-06 19:27:09.508247076 +0000 UTC m=+963.689428883"
Sep 06 19:27:27.959494 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:27.959421    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7731-140362614360368"]
Sep 06 19:27:27.963109 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:27.963091    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7731-140362614360368"
Sep 06 19:27:27.965726 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:27.965704    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zsskm\""
Sep 06 19:27:27.969109 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:27.969087    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7731-140362614360368"]
Sep 06 19:27:28.039916 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:28.039876    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtknf\" (UniqueName: \"kubernetes.io/projected/9826384d-7f13-492a-b76f-9a52702acf73-kube-api-access-rtknf\") pod \"test-curl-7731-140362614360368\" (UID: \"9826384d-7f13-492a-b76f-9a52702acf73\") " pod="opendatahub/test-curl-7731-140362614360368"
Sep 06 19:27:28.141126 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:28.141098    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtknf\" (UniqueName: \"kubernetes.io/projected/9826384d-7f13-492a-b76f-9a52702acf73-kube-api-access-rtknf\") pod \"test-curl-7731-140362614360368\" (UID: \"9826384d-7f13-492a-b76f-9a52702acf73\") " pod="opendatahub/test-curl-7731-140362614360368"
Sep 06 19:27:28.149645 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:28.149620    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtknf\" (UniqueName: \"kubernetes.io/projected/9826384d-7f13-492a-b76f-9a52702acf73-kube-api-access-rtknf\") pod \"test-curl-7731-140362614360368\" (UID: \"9826384d-7f13-492a-b76f-9a52702acf73\") " pod="opendatahub/test-curl-7731-140362614360368"
Sep 06 19:27:28.274082 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:28.274019    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7731-140362614360368"
Sep 06 19:27:28.544197 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:28.544164    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7731-140362614360368"]
Sep 06 19:27:28.545701 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:27:28.545659    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9826384d_7f13_492a_b76f_9a52702acf73.slice/crio-d1e1729f9f91027c6c114bc48076067fbcc6254fab11fc9239c851f25c3430a7 WatchSource:0}: Error finding container d1e1729f9f91027c6c114bc48076067fbcc6254fab11fc9239c851f25c3430a7: Status 404 returned error can't find the container with id d1e1729f9f91027c6c114bc48076067fbcc6254fab11fc9239c851f25c3430a7
Sep 06 19:27:28.547175 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:28.547156    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 19:27:28.592555 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:28.592527    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7731-140362614360368" event={"ID":"9826384d-7f13-492a-b76f-9a52702acf73","Type":"ContainerStarted","Data":"d1e1729f9f91027c6c114bc48076067fbcc6254fab11fc9239c851f25c3430a7"}
Sep 06 19:27:29.558952 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:29.558872    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:27:30.608059 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:30.608023    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7731-140362614360368" event={"ID":"9826384d-7f13-492a-b76f-9a52702acf73","Type":"ContainerStarted","Data":"42aea4649b564c95563d528fceda93f2e3828f883327ee24e2e829dec7613cdf"}
Sep 06 19:27:30.627760 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:30.627707    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7731-140362614360368" podStartSLOduration=1.669139259 podStartE2EDuration="3.627687935s" totalImagesPullingTime="1.958548676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:27:30.627400728 +0000 UTC m=+984.808582534" watchObservedRunningTime="2026-09-06 19:27:30.627687935 +0000 UTC m=+984.808869738"
Sep 06 19:27:30.725299 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:30.725270    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7731-140362614360368_9826384d-7f13-492a-b76f-9a52702acf73/test-curl-7731-140362614360368/0.log"
Sep 06 19:27:31.454922 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:31.454893    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh"]
Sep 06 19:27:31.458506 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:31.458490    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh"
Sep 06 19:27:31.461319 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:31.461294    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5307a7-dockercfg-2ksnh\""
Sep 06 19:27:31.462247 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:31.462222    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5307a7\""
Sep 06 19:27:31.470970 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:31.470946    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh"]
Sep 06 19:27:31.574646 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:31.574590    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73058c50-af0d-4e86-9a7c-5a897270c5b7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh\" (UID: \"73058c50-af0d-4e86-9a7c-5a897270c5b7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh"
Sep 06 19:27:31.574828 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:31.574670    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-794ff\" (UniqueName: \"kubernetes.io/projected/73058c50-af0d-4e86-9a7c-5a897270c5b7-kube-api-access-794ff\") pod \"payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh\" (UID: \"73058c50-af0d-4e86-9a7c-5a897270c5b7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh"
Sep 06 19:27:31.615376 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:31.615344    2566 generic.go:356] "Generic (PLEG): container finished" podID="9826384d-7f13-492a-b76f-9a52702acf73" containerID="42aea4649b564c95563d528fceda93f2e3828f883327ee24e2e829dec7613cdf" exitCode=0
Sep 06 19:27:31.615812 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:31.615433    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7731-140362614360368" event={"ID":"9826384d-7f13-492a-b76f-9a52702acf73","Type":"ContainerDied","Data":"42aea4649b564c95563d528fceda93f2e3828f883327ee24e2e829dec7613cdf"}
Sep 06 19:27:31.675234 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:31.675200    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73058c50-af0d-4e86-9a7c-5a897270c5b7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh\" (UID: \"73058c50-af0d-4e86-9a7c-5a897270c5b7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh"
Sep 06 19:27:31.675420 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:31.675263    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-794ff\" (UniqueName: \"kubernetes.io/projected/73058c50-af0d-4e86-9a7c-5a897270c5b7-kube-api-access-794ff\") pod \"payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh\" (UID: \"73058c50-af0d-4e86-9a7c-5a897270c5b7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh"
Sep 06 19:27:31.675970 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:31.675950    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73058c50-af0d-4e86-9a7c-5a897270c5b7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh\" (UID: \"73058c50-af0d-4e86-9a7c-5a897270c5b7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh"
Sep 06 19:27:31.683120 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:31.683097    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-794ff\" (UniqueName: \"kubernetes.io/projected/73058c50-af0d-4e86-9a7c-5a897270c5b7-kube-api-access-794ff\") pod \"payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh\" (UID: \"73058c50-af0d-4e86-9a7c-5a897270c5b7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh"
Sep 06 19:27:31.769375 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:31.769280    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh"
Sep 06 19:27:32.039345 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:32.039317    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh"]
Sep 06 19:27:32.041587 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:27:32.041564    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73058c50_af0d_4e86_9a7c_5a897270c5b7.slice/crio-a50c011bdad1a93e5c350e50816901a2d0a02a2e81104e9b27d18d6350bb7789 WatchSource:0}: Error finding container a50c011bdad1a93e5c350e50816901a2d0a02a2e81104e9b27d18d6350bb7789: Status 404 returned error can't find the container with id a50c011bdad1a93e5c350e50816901a2d0a02a2e81104e9b27d18d6350bb7789
Sep 06 19:27:32.622374 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:32.622338    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh" event={"ID":"73058c50-af0d-4e86-9a7c-5a897270c5b7","Type":"ContainerStarted","Data":"a50c011bdad1a93e5c350e50816901a2d0a02a2e81104e9b27d18d6350bb7789"}
Sep 06 19:27:32.752336 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:32.752313    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7731-140362614360368"
Sep 06 19:27:32.782974 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:32.782941    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7731-140362614360368"]
Sep 06 19:27:32.787201 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:32.787174    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7731-140362614360368"]
Sep 06 19:27:32.884708 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:32.884620    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtknf\" (UniqueName: \"kubernetes.io/projected/9826384d-7f13-492a-b76f-9a52702acf73-kube-api-access-rtknf\") pod \"9826384d-7f13-492a-b76f-9a52702acf73\" (UID: \"9826384d-7f13-492a-b76f-9a52702acf73\") "
Sep 06 19:27:32.887043 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:32.887012    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9826384d-7f13-492a-b76f-9a52702acf73-kube-api-access-rtknf" (OuterVolumeSpecName: "kube-api-access-rtknf") pod "9826384d-7f13-492a-b76f-9a52702acf73" (UID: "9826384d-7f13-492a-b76f-9a52702acf73"). InnerVolumeSpecName "kube-api-access-rtknf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:27:32.909083 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:32.909056    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7731-140362614359344"]
Sep 06 19:27:32.909585 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:32.909570    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9826384d-7f13-492a-b76f-9a52702acf73" containerName="test-curl-7731-140362614360368"
Sep 06 19:27:32.909685 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:32.909588    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9826384d-7f13-492a-b76f-9a52702acf73" containerName="test-curl-7731-140362614360368"
Sep 06 19:27:32.909753 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:32.909706    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9826384d-7f13-492a-b76f-9a52702acf73" containerName="test-curl-7731-140362614360368"
Sep 06 19:27:32.912848 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:32.912821    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7731-140362614359344"
Sep 06 19:27:32.916075 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:32.916051    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7731-140362614359344"]
Sep 06 19:27:32.985268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:32.985239    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xqpt\" (UniqueName: \"kubernetes.io/projected/0f0895a7-503e-4069-9897-cc4e1be761a6-kube-api-access-9xqpt\") pod \"test-curl-7731-140362614359344\" (UID: \"0f0895a7-503e-4069-9897-cc4e1be761a6\") " pod="opendatahub/test-curl-7731-140362614359344"
Sep 06 19:27:32.985417 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:32.985291    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtknf\" (UniqueName: \"kubernetes.io/projected/9826384d-7f13-492a-b76f-9a52702acf73-kube-api-access-rtknf\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:27:33.087026 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:33.086989    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xqpt\" (UniqueName: \"kubernetes.io/projected/0f0895a7-503e-4069-9897-cc4e1be761a6-kube-api-access-9xqpt\") pod \"test-curl-7731-140362614359344\" (UID: \"0f0895a7-503e-4069-9897-cc4e1be761a6\") " pod="opendatahub/test-curl-7731-140362614359344"
Sep 06 19:27:33.096058 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:33.096026    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xqpt\" (UniqueName: \"kubernetes.io/projected/0f0895a7-503e-4069-9897-cc4e1be761a6-kube-api-access-9xqpt\") pod \"test-curl-7731-140362614359344\" (UID: \"0f0895a7-503e-4069-9897-cc4e1be761a6\") " pod="opendatahub/test-curl-7731-140362614359344"
Sep 06 19:27:33.225506 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:33.225474    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7731-140362614359344"
Sep 06 19:27:33.509567 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:33.509544    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7731-140362614359344"]
Sep 06 19:27:33.511785 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:27:33.511760    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f0895a7_503e_4069_9897_cc4e1be761a6.slice/crio-23d0d58c0d3f9e968fcdaac047acb47c5e7274920132521de6f68667ba1a6f22 WatchSource:0}: Error finding container 23d0d58c0d3f9e968fcdaac047acb47c5e7274920132521de6f68667ba1a6f22: Status 404 returned error can't find the container with id 23d0d58c0d3f9e968fcdaac047acb47c5e7274920132521de6f68667ba1a6f22
Sep 06 19:27:33.629724 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:33.629699    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7731-140362614360368"
Sep 06 19:27:33.630159 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:33.629707    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1e1729f9f91027c6c114bc48076067fbcc6254fab11fc9239c851f25c3430a7"
Sep 06 19:27:33.630980 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:33.630948    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7731-140362614359344" event={"ID":"0f0895a7-503e-4069-9897-cc4e1be761a6","Type":"ContainerStarted","Data":"23d0d58c0d3f9e968fcdaac047acb47c5e7274920132521de6f68667ba1a6f22"}
Sep 06 19:27:33.640127 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:27:33.640102    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7731-140362614360368\" is forbidden: User \"system:node:ip-10-0-129-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-252.ec2.internal' and this object" podUID="9826384d-7f13-492a-b76f-9a52702acf73" pod="opendatahub/test-curl-7731-140362614360368"
Sep 06 19:27:34.312152 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:34.312123    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9826384d-7f13-492a-b76f-9a52702acf73" path="/var/lib/kubelet/pods/9826384d-7f13-492a-b76f-9a52702acf73/volumes"
Sep 06 19:27:34.644196 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:34.644088    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh" event={"ID":"73058c50-af0d-4e86-9a7c-5a897270c5b7","Type":"ContainerStarted","Data":"4b7ddbc0ce5b279d107b1e25e846f505bc895fd79b655b139b321d2131981226"}
Sep 06 19:27:34.644196 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:34.644187    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh"
Sep 06 19:27:34.646127 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:34.646098    2566 generic.go:356] "Generic (PLEG): container finished" podID="0f0895a7-503e-4069-9897-cc4e1be761a6" containerID="b3d8a6dd755e791bc259bbf707c2db67729901a7e51a4aca35d95242e44f2d77" exitCode=0
Sep 06 19:27:34.646280 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:34.646183    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7731-140362614359344" event={"ID":"0f0895a7-503e-4069-9897-cc4e1be761a6","Type":"ContainerDied","Data":"b3d8a6dd755e791bc259bbf707c2db67729901a7e51a4aca35d95242e44f2d77"}
Sep 06 19:27:34.659882 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:34.659832    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh" podStartSLOduration=1.712733605 podStartE2EDuration="3.659818646s" totalImagesPullingTime="1.947085041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:27:34.658470809 +0000 UTC m=+988.839652609" watchObservedRunningTime="2026-09-06 19:27:34.659818646 +0000 UTC m=+988.841000452"
Sep 06 19:27:35.788240 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:35.788211    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7731-140362614359344"
Sep 06 19:27:35.818248 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:35.818221    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7731-140362614359344_0f0895a7-503e-4069-9897-cc4e1be761a6/test-curl-7731-140362614359344/0.log"
Sep 06 19:27:35.842871 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:35.842836    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7731-140362614359344"]
Sep 06 19:27:35.851157 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:35.851131    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7731-140362614359344"]
Sep 06 19:27:35.912978 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:35.912901    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xqpt\" (UniqueName: \"kubernetes.io/projected/0f0895a7-503e-4069-9897-cc4e1be761a6-kube-api-access-9xqpt\") pod \"0f0895a7-503e-4069-9897-cc4e1be761a6\" (UID: \"0f0895a7-503e-4069-9897-cc4e1be761a6\") "
Sep 06 19:27:35.914955 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:35.914928    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f0895a7-503e-4069-9897-cc4e1be761a6-kube-api-access-9xqpt" (OuterVolumeSpecName: "kube-api-access-9xqpt") pod "0f0895a7-503e-4069-9897-cc4e1be761a6" (UID: "0f0895a7-503e-4069-9897-cc4e1be761a6"). InnerVolumeSpecName "kube-api-access-9xqpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:27:36.013739 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:36.013705    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xqpt\" (UniqueName: \"kubernetes.io/projected/0f0895a7-503e-4069-9897-cc4e1be761a6-kube-api-access-9xqpt\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:27:36.312358 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:36.312322    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f0895a7-503e-4069-9897-cc4e1be761a6" path="/var/lib/kubelet/pods/0f0895a7-503e-4069-9897-cc4e1be761a6/volumes"
Sep 06 19:27:36.660505 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:36.660432    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7731-140362614359344"
Sep 06 19:27:36.660505 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:36.660487    2566 scope.go:117] "RemoveContainer" containerID="b3d8a6dd755e791bc259bbf707c2db67729901a7e51a4aca35d95242e44f2d77"
Sep 06 19:27:38.831987 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:38.831956    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l"]
Sep 06 19:27:38.832318 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:38.832304    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f0895a7-503e-4069-9897-cc4e1be761a6" containerName="test-curl-7731-140362614359344"
Sep 06 19:27:38.832318 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:38.832315    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f0895a7-503e-4069-9897-cc4e1be761a6" containerName="test-curl-7731-140362614359344"
Sep 06 19:27:38.832399 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:38.832380    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f0895a7-503e-4069-9897-cc4e1be761a6" containerName="test-curl-7731-140362614359344"
Sep 06 19:27:38.836945 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:38.836925    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l"
Sep 06 19:27:38.839261 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:38.839245    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0281bd63\""
Sep 06 19:27:38.847341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:38.847320    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l"]
Sep 06 19:27:38.938532 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:38.938499    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-955xr\" (UniqueName: \"kubernetes.io/projected/f630ed4d-98ec-4176-b123-e7963dda5608-kube-api-access-955xr\") pod \"payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l\" (UID: \"f630ed4d-98ec-4176-b123-e7963dda5608\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l"
Sep 06 19:27:38.938694 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:38.938546    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f630ed4d-98ec-4176-b123-e7963dda5608-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l\" (UID: \"f630ed4d-98ec-4176-b123-e7963dda5608\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l"
Sep 06 19:27:39.039776 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:39.039738    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-955xr\" (UniqueName: \"kubernetes.io/projected/f630ed4d-98ec-4176-b123-e7963dda5608-kube-api-access-955xr\") pod \"payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l\" (UID: \"f630ed4d-98ec-4176-b123-e7963dda5608\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l"
Sep 06 19:27:39.039776 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:39.039777    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f630ed4d-98ec-4176-b123-e7963dda5608-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l\" (UID: \"f630ed4d-98ec-4176-b123-e7963dda5608\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l"
Sep 06 19:27:39.040383 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:39.040364    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f630ed4d-98ec-4176-b123-e7963dda5608-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l\" (UID: \"f630ed4d-98ec-4176-b123-e7963dda5608\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l"
Sep 06 19:27:39.047654 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:39.047630    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-955xr\" (UniqueName: \"kubernetes.io/projected/f630ed4d-98ec-4176-b123-e7963dda5608-kube-api-access-955xr\") pod \"payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l\" (UID: \"f630ed4d-98ec-4176-b123-e7963dda5608\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l"
Sep 06 19:27:39.147499 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:39.147423    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l"
Sep 06 19:27:39.418486 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:39.418460    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l"]
Sep 06 19:27:39.419799 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:27:39.419768    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf630ed4d_98ec_4176_b123_e7963dda5608.slice/crio-25c6808d47fa875f89b796b00e82e916aefc29f8a7c8fb3eaf1769cdfc3b3c1e WatchSource:0}: Error finding container 25c6808d47fa875f89b796b00e82e916aefc29f8a7c8fb3eaf1769cdfc3b3c1e: Status 404 returned error can't find the container with id 25c6808d47fa875f89b796b00e82e916aefc29f8a7c8fb3eaf1769cdfc3b3c1e
Sep 06 19:27:39.678723 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:39.678643    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l" event={"ID":"f630ed4d-98ec-4176-b123-e7963dda5608","Type":"ContainerStarted","Data":"eb62ec484d0053836f9b68f847bcfa8fe91b5544861c897d03fdea3d3fec6039"}
Sep 06 19:27:39.678723 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:39.678700    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l" event={"ID":"f630ed4d-98ec-4176-b123-e7963dda5608","Type":"ContainerStarted","Data":"25c6808d47fa875f89b796b00e82e916aefc29f8a7c8fb3eaf1769cdfc3b3c1e"}
Sep 06 19:27:39.678935 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:39.678853    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l"
Sep 06 19:27:39.695620 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:39.695563    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l" podStartSLOduration=1.69555082 podStartE2EDuration="1.69555082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:27:39.692988509 +0000 UTC m=+993.874170316" watchObservedRunningTime="2026-09-06 19:27:39.69555082 +0000 UTC m=+993.876732628"
Sep 06 19:27:41.332740 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.332703    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff"]
Sep 06 19:27:41.336329 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.336310    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff"
Sep 06 19:27:41.338735 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.338715    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-21b01e-dockercfg-mzg5l\""
Sep 06 19:27:41.338735 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.338724    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-21b01e\""
Sep 06 19:27:41.349330 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.349312    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff"]
Sep 06 19:27:41.377913 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.377880    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp"]
Sep 06 19:27:41.381298 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.381284    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp"
Sep 06 19:27:41.388890 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.388872    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp"]
Sep 06 19:27:41.459814 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.459771    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ce5aa5-2753-429d-bfb0-1493556b5999-plugins-config-volume\") pod \"payload-processing-e2e-autores-21b01e-6674886df9-dd6kp\" (UID: \"29ce5aa5-2753-429d-bfb0-1493556b5999\") " pod="openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp"
Sep 06 19:27:41.459995 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.459885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g27xh\" (UniqueName: \"kubernetes.io/projected/b631e3c5-9c83-47ea-88d7-b02208086dcf-kube-api-access-g27xh\") pod \"payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff\" (UID: \"b631e3c5-9c83-47ea-88d7-b02208086dcf\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff"
Sep 06 19:27:41.459995 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.459921    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b631e3c5-9c83-47ea-88d7-b02208086dcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff\" (UID: \"b631e3c5-9c83-47ea-88d7-b02208086dcf\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff"
Sep 06 19:27:41.459995 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.459956    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7fxr\" (UniqueName: \"kubernetes.io/projected/29ce5aa5-2753-429d-bfb0-1493556b5999-kube-api-access-q7fxr\") pod \"payload-processing-e2e-autores-21b01e-6674886df9-dd6kp\" (UID: \"29ce5aa5-2753-429d-bfb0-1493556b5999\") " pod="openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp"
Sep 06 19:27:41.560466 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.560434    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ce5aa5-2753-429d-bfb0-1493556b5999-plugins-config-volume\") pod \"payload-processing-e2e-autores-21b01e-6674886df9-dd6kp\" (UID: \"29ce5aa5-2753-429d-bfb0-1493556b5999\") " pod="openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp"
Sep 06 19:27:41.560629 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.560512    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g27xh\" (UniqueName: \"kubernetes.io/projected/b631e3c5-9c83-47ea-88d7-b02208086dcf-kube-api-access-g27xh\") pod \"payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff\" (UID: \"b631e3c5-9c83-47ea-88d7-b02208086dcf\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff"
Sep 06 19:27:41.560629 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.560545    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b631e3c5-9c83-47ea-88d7-b02208086dcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff\" (UID: \"b631e3c5-9c83-47ea-88d7-b02208086dcf\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff"
Sep 06 19:27:41.560629 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.560580    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7fxr\" (UniqueName: \"kubernetes.io/projected/29ce5aa5-2753-429d-bfb0-1493556b5999-kube-api-access-q7fxr\") pod \"payload-processing-e2e-autores-21b01e-6674886df9-dd6kp\" (UID: \"29ce5aa5-2753-429d-bfb0-1493556b5999\") " pod="openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp"
Sep 06 19:27:41.561169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.561151    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b631e3c5-9c83-47ea-88d7-b02208086dcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff\" (UID: \"b631e3c5-9c83-47ea-88d7-b02208086dcf\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff"
Sep 06 19:27:41.561232 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.561217    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ce5aa5-2753-429d-bfb0-1493556b5999-plugins-config-volume\") pod \"payload-processing-e2e-autores-21b01e-6674886df9-dd6kp\" (UID: \"29ce5aa5-2753-429d-bfb0-1493556b5999\") " pod="openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp"
Sep 06 19:27:41.568736 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.568716    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7fxr\" (UniqueName: \"kubernetes.io/projected/29ce5aa5-2753-429d-bfb0-1493556b5999-kube-api-access-q7fxr\") pod \"payload-processing-e2e-autores-21b01e-6674886df9-dd6kp\" (UID: \"29ce5aa5-2753-429d-bfb0-1493556b5999\") " pod="openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp"
Sep 06 19:27:41.568818 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.568777    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g27xh\" (UniqueName: \"kubernetes.io/projected/b631e3c5-9c83-47ea-88d7-b02208086dcf-kube-api-access-g27xh\") pod \"payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff\" (UID: \"b631e3c5-9c83-47ea-88d7-b02208086dcf\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff"
Sep 06 19:27:41.646817 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.646747    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff"
Sep 06 19:27:41.692139 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.692108    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp"
Sep 06 19:27:41.917127 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.917095    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff"]
Sep 06 19:27:41.920150 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:27:41.920116    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb631e3c5_9c83_47ea_88d7_b02208086dcf.slice/crio-7932a14af776395e10732fb3334ae57c76ddd0bda393e831a87c16f52821469a WatchSource:0}: Error finding container 7932a14af776395e10732fb3334ae57c76ddd0bda393e831a87c16f52821469a: Status 404 returned error can't find the container with id 7932a14af776395e10732fb3334ae57c76ddd0bda393e831a87c16f52821469a
Sep 06 19:27:41.966871 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:41.966846    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp"]
Sep 06 19:27:41.968091 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:27:41.968063    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ce5aa5_2753_429d_bfb0_1493556b5999.slice/crio-3fcbe48f8308c1aa0aac84dcfb46c5b4a800523f583ff76e8a8add8952769d0a WatchSource:0}: Error finding container 3fcbe48f8308c1aa0aac84dcfb46c5b4a800523f583ff76e8a8add8952769d0a: Status 404 returned error can't find the container with id 3fcbe48f8308c1aa0aac84dcfb46c5b4a800523f583ff76e8a8add8952769d0a
Sep 06 19:27:42.704947 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:42.704894    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp" event={"ID":"29ce5aa5-2753-429d-bfb0-1493556b5999","Type":"ContainerStarted","Data":"a95b50a9972880f825016b7a267a976bdf34f2155b4dc085f4fd64f59ee7db89"}
Sep 06 19:27:42.705419 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:42.705089    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp"
Sep 06 19:27:42.705419 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:42.705116    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp" event={"ID":"29ce5aa5-2753-429d-bfb0-1493556b5999","Type":"ContainerStarted","Data":"3fcbe48f8308c1aa0aac84dcfb46c5b4a800523f583ff76e8a8add8952769d0a"}
Sep 06 19:27:42.706579 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:42.706556    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff" event={"ID":"b631e3c5-9c83-47ea-88d7-b02208086dcf","Type":"ContainerStarted","Data":"82f65287ce141b97a73b8c9597a1f90db6e81739d73ae8e7945db28f5fe2a7a0"}
Sep 06 19:27:42.706710 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:42.706595    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff" event={"ID":"b631e3c5-9c83-47ea-88d7-b02208086dcf","Type":"ContainerStarted","Data":"7932a14af776395e10732fb3334ae57c76ddd0bda393e831a87c16f52821469a"}
Sep 06 19:27:42.706710 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:42.706645    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff"
Sep 06 19:27:42.721681 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:42.721641    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp" podStartSLOduration=1.7216291529999999 podStartE2EDuration="1.721629153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:27:42.719656804 +0000 UTC m=+996.900838609" watchObservedRunningTime="2026-09-06 19:27:42.721629153 +0000 UTC m=+996.902810991"
Sep 06 19:27:42.735727 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:42.735691    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff" podStartSLOduration=1.735679561 podStartE2EDuration="1.735679561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:27:42.73335947 +0000 UTC m=+996.914541288" watchObservedRunningTime="2026-09-06 19:27:42.735679561 +0000 UTC m=+996.916861399"
Sep 06 19:27:43.161127 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:43.161038    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:27:45.656417 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:45.656386    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh"
Sep 06 19:27:50.688536 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:50.688501    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l"
Sep 06 19:27:53.714573 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:53.714538    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp"
Sep 06 19:27:53.715004 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:53.714831    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff"
Sep 06 19:27:58.867424 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:58.867391    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"]
Sep 06 19:27:58.872101 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:58.872078    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"
Sep 06 19:27:58.874664 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:58.874627    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d73678-serving-cert\""
Sep 06 19:27:58.874803 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:58.874682    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gfj4k\""
Sep 06 19:27:58.875338 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:58.875314    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 06 19:27:58.875446 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:58.875373    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 19:27:58.875689 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:58.875664    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 06 19:27:58.882826 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:58.882805    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"]
Sep 06 19:27:58.909576 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:58.909553    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c493a985-9a3a-45cb-a5a3-db914fde1e6b-maas-api-tls\") pod \"maas-api-e2e-inf-b-d73678-847cbc69f-szf5j\" (UID: \"c493a985-9a3a-45cb-a5a3-db914fde1e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"
Sep 06 19:27:58.909715 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:58.909630    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c493a985-9a3a-45cb-a5a3-db914fde1e6b-metrics-certs\") pod \"maas-api-e2e-inf-b-d73678-847cbc69f-szf5j\" (UID: \"c493a985-9a3a-45cb-a5a3-db914fde1e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"
Sep 06 19:27:58.909715 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:58.909681    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c493a985-9a3a-45cb-a5a3-db914fde1e6b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d73678-847cbc69f-szf5j\" (UID: \"c493a985-9a3a-45cb-a5a3-db914fde1e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"
Sep 06 19:27:58.909715 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:58.909706    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nd4p\" (UniqueName: \"kubernetes.io/projected/c493a985-9a3a-45cb-a5a3-db914fde1e6b-kube-api-access-9nd4p\") pod \"maas-api-e2e-inf-b-d73678-847cbc69f-szf5j\" (UID: \"c493a985-9a3a-45cb-a5a3-db914fde1e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"
Sep 06 19:27:59.010753 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.010726    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c493a985-9a3a-45cb-a5a3-db914fde1e6b-metrics-certs\") pod \"maas-api-e2e-inf-b-d73678-847cbc69f-szf5j\" (UID: \"c493a985-9a3a-45cb-a5a3-db914fde1e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"
Sep 06 19:27:59.011054 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.010779    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c493a985-9a3a-45cb-a5a3-db914fde1e6b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d73678-847cbc69f-szf5j\" (UID: \"c493a985-9a3a-45cb-a5a3-db914fde1e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"
Sep 06 19:27:59.011054 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.010824    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nd4p\" (UniqueName: \"kubernetes.io/projected/c493a985-9a3a-45cb-a5a3-db914fde1e6b-kube-api-access-9nd4p\") pod \"maas-api-e2e-inf-b-d73678-847cbc69f-szf5j\" (UID: \"c493a985-9a3a-45cb-a5a3-db914fde1e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"
Sep 06 19:27:59.011054 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.010858    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c493a985-9a3a-45cb-a5a3-db914fde1e6b-maas-api-tls\") pod \"maas-api-e2e-inf-b-d73678-847cbc69f-szf5j\" (UID: \"c493a985-9a3a-45cb-a5a3-db914fde1e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"
Sep 06 19:27:59.011054 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:27:59.010990    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-serving-cert: secret "maas-api-e2e-inf-b-d73678-serving-cert" not found
Sep 06 19:27:59.011343 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:27:59.011073    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c493a985-9a3a-45cb-a5a3-db914fde1e6b-maas-api-tls podName:c493a985-9a3a-45cb-a5a3-db914fde1e6b nodeName:}" failed. No retries permitted until 2026-09-06 19:27:59.511052401 +0000 UTC m=+1013.692234199 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c493a985-9a3a-45cb-a5a3-db914fde1e6b-maas-api-tls") pod "maas-api-e2e-inf-b-d73678-847cbc69f-szf5j" (UID: "c493a985-9a3a-45cb-a5a3-db914fde1e6b") : secret "maas-api-e2e-inf-b-d73678-serving-cert" not found
Sep 06 19:27:59.011626 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.011589    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c493a985-9a3a-45cb-a5a3-db914fde1e6b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d73678-847cbc69f-szf5j\" (UID: \"c493a985-9a3a-45cb-a5a3-db914fde1e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"
Sep 06 19:27:59.013306 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.013282    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c493a985-9a3a-45cb-a5a3-db914fde1e6b-metrics-certs\") pod \"maas-api-e2e-inf-b-d73678-847cbc69f-szf5j\" (UID: \"c493a985-9a3a-45cb-a5a3-db914fde1e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"
Sep 06 19:27:59.020294 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.020263    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nd4p\" (UniqueName: \"kubernetes.io/projected/c493a985-9a3a-45cb-a5a3-db914fde1e6b-kube-api-access-9nd4p\") pod \"maas-api-e2e-inf-b-d73678-847cbc69f-szf5j\" (UID: \"c493a985-9a3a-45cb-a5a3-db914fde1e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"
Sep 06 19:27:59.471633 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.471583    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:27:59.475768 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.475747    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x"]
Sep 06 19:27:59.479489 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.479467    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x"
Sep 06 19:27:59.482174 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.482149    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d73678-dockercfg-gxjg4\""
Sep 06 19:27:59.482270 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.482236    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d73678\""
Sep 06 19:27:59.489859 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.489833    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x"]
Sep 06 19:27:59.509153 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.509131    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf"]
Sep 06 19:27:59.512584 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.512569    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf"
Sep 06 19:27:59.516285 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.516259    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f68d30-5da0-450e-b18c-ddf97c6e4028-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x\" (UID: \"b5f68d30-5da0-450e-b18c-ddf97c6e4028\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x"
Sep 06 19:27:59.516392 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.516336    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72t6r\" (UniqueName: \"kubernetes.io/projected/33e078b2-1619-4b07-90c6-17cb2fe2edf1-kube-api-access-72t6r\") pod \"payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf\" (UID: \"33e078b2-1619-4b07-90c6-17cb2fe2edf1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf"
Sep 06 19:27:59.516465 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.516383    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srtzq\" (UniqueName: \"kubernetes.io/projected/b5f68d30-5da0-450e-b18c-ddf97c6e4028-kube-api-access-srtzq\") pod \"payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x\" (UID: \"b5f68d30-5da0-450e-b18c-ddf97c6e4028\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x"
Sep 06 19:27:59.516465 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.516438    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c493a985-9a3a-45cb-a5a3-db914fde1e6b-maas-api-tls\") pod \"maas-api-e2e-inf-b-d73678-847cbc69f-szf5j\" (UID: \"c493a985-9a3a-45cb-a5a3-db914fde1e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"
Sep 06 19:27:59.516587 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.516480    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33e078b2-1619-4b07-90c6-17cb2fe2edf1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf\" (UID: \"33e078b2-1619-4b07-90c6-17cb2fe2edf1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf"
Sep 06 19:27:59.519442 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.519417    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c493a985-9a3a-45cb-a5a3-db914fde1e6b-maas-api-tls\") pod \"maas-api-e2e-inf-b-d73678-847cbc69f-szf5j\" (UID: \"c493a985-9a3a-45cb-a5a3-db914fde1e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"
Sep 06 19:27:59.523644 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.523597    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf"]
Sep 06 19:27:59.617142 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.617102    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33e078b2-1619-4b07-90c6-17cb2fe2edf1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf\" (UID: \"33e078b2-1619-4b07-90c6-17cb2fe2edf1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf"
Sep 06 19:27:59.617328 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.617203    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f68d30-5da0-450e-b18c-ddf97c6e4028-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x\" (UID: \"b5f68d30-5da0-450e-b18c-ddf97c6e4028\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x"
Sep 06 19:27:59.617328 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.617230    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72t6r\" (UniqueName: \"kubernetes.io/projected/33e078b2-1619-4b07-90c6-17cb2fe2edf1-kube-api-access-72t6r\") pod \"payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf\" (UID: \"33e078b2-1619-4b07-90c6-17cb2fe2edf1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf"
Sep 06 19:27:59.617328 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.617246    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srtzq\" (UniqueName: \"kubernetes.io/projected/b5f68d30-5da0-450e-b18c-ddf97c6e4028-kube-api-access-srtzq\") pod \"payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x\" (UID: \"b5f68d30-5da0-450e-b18c-ddf97c6e4028\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x"
Sep 06 19:27:59.617821 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.617795    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33e078b2-1619-4b07-90c6-17cb2fe2edf1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf\" (UID: \"33e078b2-1619-4b07-90c6-17cb2fe2edf1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf"
Sep 06 19:27:59.618025 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.618003    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f68d30-5da0-450e-b18c-ddf97c6e4028-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x\" (UID: \"b5f68d30-5da0-450e-b18c-ddf97c6e4028\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x"
Sep 06 19:27:59.629815 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.629789    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72t6r\" (UniqueName: \"kubernetes.io/projected/33e078b2-1619-4b07-90c6-17cb2fe2edf1-kube-api-access-72t6r\") pod \"payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf\" (UID: \"33e078b2-1619-4b07-90c6-17cb2fe2edf1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf"
Sep 06 19:27:59.629815 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.629799    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srtzq\" (UniqueName: \"kubernetes.io/projected/b5f68d30-5da0-450e-b18c-ddf97c6e4028-kube-api-access-srtzq\") pod \"payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x\" (UID: \"b5f68d30-5da0-450e-b18c-ddf97c6e4028\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x"
Sep 06 19:27:59.786298 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.786219    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"
Sep 06 19:27:59.794345 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.794281    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x"
Sep 06 19:27:59.832737 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:27:59.832704    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf"
Sep 06 19:28:00.071151 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:00.071033    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x"]
Sep 06 19:28:00.072919 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:28:00.072892    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f68d30_5da0_450e_b18c_ddf97c6e4028.slice/crio-d13c947533b1dda8a243729f2c128ede302d5250a0ab62ddc3109b1e9c9be621 WatchSource:0}: Error finding container d13c947533b1dda8a243729f2c128ede302d5250a0ab62ddc3109b1e9c9be621: Status 404 returned error can't find the container with id d13c947533b1dda8a243729f2c128ede302d5250a0ab62ddc3109b1e9c9be621
Sep 06 19:28:00.090294 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:00.090266    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"]
Sep 06 19:28:00.091955 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:28:00.091930    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc493a985_9a3a_45cb_a5a3_db914fde1e6b.slice/crio-8037042283c412c51c654feb08313209ce1a495c116715ac32c1af49319032c2 WatchSource:0}: Error finding container 8037042283c412c51c654feb08313209ce1a495c116715ac32c1af49319032c2: Status 404 returned error can't find the container with id 8037042283c412c51c654feb08313209ce1a495c116715ac32c1af49319032c2
Sep 06 19:28:00.134236 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:00.134215    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf"]
Sep 06 19:28:00.142253 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:28:00.142156    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e078b2_1619_4b07_90c6_17cb2fe2edf1.slice/crio-b10d2963d3225f70678d824e86cd136dac5f53f0f879c70b163d79304b55363a WatchSource:0}: Error finding container b10d2963d3225f70678d824e86cd136dac5f53f0f879c70b163d79304b55363a: Status 404 returned error can't find the container with id b10d2963d3225f70678d824e86cd136dac5f53f0f879c70b163d79304b55363a
Sep 06 19:28:00.809959 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:00.809827    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x" event={"ID":"b5f68d30-5da0-450e-b18c-ddf97c6e4028","Type":"ContainerStarted","Data":"1adfb56e22c3a2ebd85a5d646c37a09d23c4d155671a40b3a9f6ac51c31759ff"}
Sep 06 19:28:00.809959 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:00.809897    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x"
Sep 06 19:28:00.809959 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:00.809919    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x" event={"ID":"b5f68d30-5da0-450e-b18c-ddf97c6e4028","Type":"ContainerStarted","Data":"d13c947533b1dda8a243729f2c128ede302d5250a0ab62ddc3109b1e9c9be621"}
Sep 06 19:28:00.812309 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:00.812261    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf" event={"ID":"33e078b2-1619-4b07-90c6-17cb2fe2edf1","Type":"ContainerStarted","Data":"c2a862d682748bd8e5ce4fc0457b4c70664f7b26aaff9f6b180332edee5be4d1"}
Sep 06 19:28:00.812439 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:00.812317    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf" event={"ID":"33e078b2-1619-4b07-90c6-17cb2fe2edf1","Type":"ContainerStarted","Data":"b10d2963d3225f70678d824e86cd136dac5f53f0f879c70b163d79304b55363a"}
Sep 06 19:28:00.812540 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:00.812522    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf"
Sep 06 19:28:00.814169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:00.814141    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j" event={"ID":"c493a985-9a3a-45cb-a5a3-db914fde1e6b","Type":"ContainerStarted","Data":"8037042283c412c51c654feb08313209ce1a495c116715ac32c1af49319032c2"}
Sep 06 19:28:00.831021 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:00.830966    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x" podStartSLOduration=1.830948889 podStartE2EDuration="1.830948889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:28:00.83053917 +0000 UTC m=+1015.011720976" watchObservedRunningTime="2026-09-06 19:28:00.830948889 +0000 UTC m=+1015.012130693"
Sep 06 19:28:00.858575 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:00.858528    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf" podStartSLOduration=1.8585105469999998 podStartE2EDuration="1.858510547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:28:00.855891379 +0000 UTC m=+1015.037073183" watchObservedRunningTime="2026-09-06 19:28:00.858510547 +0000 UTC m=+1015.039692354"
Sep 06 19:28:01.821740 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:01.821643    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j" event={"ID":"c493a985-9a3a-45cb-a5a3-db914fde1e6b","Type":"ContainerStarted","Data":"c7127f8f1d0a7cd32cc90dabd7f7cd5166ec6cb42244c4a602a5c0ddef58187e"}
Sep 06 19:28:01.822152 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:01.821944    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"
Sep 06 19:28:01.842322 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:01.842277    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j" podStartSLOduration=2.420371366 podStartE2EDuration="3.842263458s" totalImagesPullingTime="1.421892092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:27:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:28:01.840109455 +0000 UTC m=+1016.021291260" watchObservedRunningTime="2026-09-06 19:28:01.842263458 +0000 UTC m=+1016.023445263"
Sep 06 19:28:07.833468 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:07.833427    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"
Sep 06 19:28:11.823301 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:11.823270    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf"
Sep 06 19:28:11.824057 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:11.824039    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x"
Sep 06 19:28:13.050446 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:13.050408    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:28:19.049246 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:19.049214    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l"]
Sep 06 19:28:19.049737 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:19.049451    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l" podUID="f630ed4d-98ec-4176-b123-e7963dda5608" containerName="payload-pre-processing" containerID="cri-o://eb62ec484d0053836f9b68f847bcfa8fe91b5544861c897d03fdea3d3fec6039" gracePeriod=30
Sep 06 19:28:19.106417 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:28:19.106388    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf630ed4d_98ec_4176_b123_e7963dda5608.slice/crio-conmon-eb62ec484d0053836f9b68f847bcfa8fe91b5544861c897d03fdea3d3fec6039.scope\": RecentStats: unable to find data in memory cache]"
Sep 06 19:28:19.303637 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:19.303556    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l"
Sep 06 19:28:19.395093 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:19.395065    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-955xr\" (UniqueName: \"kubernetes.io/projected/f630ed4d-98ec-4176-b123-e7963dda5608-kube-api-access-955xr\") pod \"f630ed4d-98ec-4176-b123-e7963dda5608\" (UID: \"f630ed4d-98ec-4176-b123-e7963dda5608\") "
Sep 06 19:28:19.395228 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:19.395114    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f630ed4d-98ec-4176-b123-e7963dda5608-plugins-config-volume\") pod \"f630ed4d-98ec-4176-b123-e7963dda5608\" (UID: \"f630ed4d-98ec-4176-b123-e7963dda5608\") "
Sep 06 19:28:19.395500 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:19.395471    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f630ed4d-98ec-4176-b123-e7963dda5608-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f630ed4d-98ec-4176-b123-e7963dda5608" (UID: "f630ed4d-98ec-4176-b123-e7963dda5608"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:28:19.397280 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:19.397257    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f630ed4d-98ec-4176-b123-e7963dda5608-kube-api-access-955xr" (OuterVolumeSpecName: "kube-api-access-955xr") pod "f630ed4d-98ec-4176-b123-e7963dda5608" (UID: "f630ed4d-98ec-4176-b123-e7963dda5608"). InnerVolumeSpecName "kube-api-access-955xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:28:19.496093 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:19.496066    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-955xr\" (UniqueName: \"kubernetes.io/projected/f630ed4d-98ec-4176-b123-e7963dda5608-kube-api-access-955xr\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:28:19.496093 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:19.496089    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f630ed4d-98ec-4176-b123-e7963dda5608-plugins-config-volume\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:28:19.921224 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:19.921189    2566 generic.go:356] "Generic (PLEG): container finished" podID="f630ed4d-98ec-4176-b123-e7963dda5608" containerID="eb62ec484d0053836f9b68f847bcfa8fe91b5544861c897d03fdea3d3fec6039" exitCode=0
Sep 06 19:28:19.921459 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:19.921234    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l"
Sep 06 19:28:19.921459 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:19.921275    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l" event={"ID":"f630ed4d-98ec-4176-b123-e7963dda5608","Type":"ContainerDied","Data":"eb62ec484d0053836f9b68f847bcfa8fe91b5544861c897d03fdea3d3fec6039"}
Sep 06 19:28:19.921459 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:19.921335    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l" event={"ID":"f630ed4d-98ec-4176-b123-e7963dda5608","Type":"ContainerDied","Data":"25c6808d47fa875f89b796b00e82e916aefc29f8a7c8fb3eaf1769cdfc3b3c1e"}
Sep 06 19:28:19.921459 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:19.921364    2566 scope.go:117] "RemoveContainer" containerID="eb62ec484d0053836f9b68f847bcfa8fe91b5544861c897d03fdea3d3fec6039"
Sep 06 19:28:19.931680 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:19.931661    2566 scope.go:117] "RemoveContainer" containerID="eb62ec484d0053836f9b68f847bcfa8fe91b5544861c897d03fdea3d3fec6039"
Sep 06 19:28:19.931970 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:28:19.931951    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb62ec484d0053836f9b68f847bcfa8fe91b5544861c897d03fdea3d3fec6039\": container with ID starting with eb62ec484d0053836f9b68f847bcfa8fe91b5544861c897d03fdea3d3fec6039 not found: ID does not exist" containerID="eb62ec484d0053836f9b68f847bcfa8fe91b5544861c897d03fdea3d3fec6039"
Sep 06 19:28:19.932031 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:19.931981    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb62ec484d0053836f9b68f847bcfa8fe91b5544861c897d03fdea3d3fec6039"} err="failed to get container status \"eb62ec484d0053836f9b68f847bcfa8fe91b5544861c897d03fdea3d3fec6039\": rpc error: code = NotFound desc = could not find container \"eb62ec484d0053836f9b68f847bcfa8fe91b5544861c897d03fdea3d3fec6039\": container with ID starting with eb62ec484d0053836f9b68f847bcfa8fe91b5544861c897d03fdea3d3fec6039 not found: ID does not exist"
Sep 06 19:28:19.949328 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:19.949304    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l"]
Sep 06 19:28:19.955482 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:19.955464    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0281bd63-96f995bbb-rkd8l"]
Sep 06 19:28:20.311686 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:20.311649    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f630ed4d-98ec-4176-b123-e7963dda5608" path="/var/lib/kubelet/pods/f630ed4d-98ec-4176-b123-e7963dda5608/volumes"
Sep 06 19:28:23.521706 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:23.521671    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt"]
Sep 06 19:28:23.522261 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:23.522241    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f630ed4d-98ec-4176-b123-e7963dda5608" containerName="payload-pre-processing"
Sep 06 19:28:23.522353 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:23.522265    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f630ed4d-98ec-4176-b123-e7963dda5608" containerName="payload-pre-processing"
Sep 06 19:28:23.522408 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:23.522401    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f630ed4d-98ec-4176-b123-e7963dda5608" containerName="payload-pre-processing"
Sep 06 19:28:23.526993 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:23.526971    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt"
Sep 06 19:28:23.529508 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:23.529484    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-053eb1b8-dockercfg-l5cnj\""
Sep 06 19:28:23.529667 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:23.529591    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-053eb1b8\""
Sep 06 19:28:23.536443 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:23.536420    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt"]
Sep 06 19:28:23.630976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:23.630945    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgpc6\" (UniqueName: \"kubernetes.io/projected/6626becc-b3d3-4c4b-b722-cb41e0e1e1d8-kube-api-access-dgpc6\") pod \"payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt\" (UID: \"6626becc-b3d3-4c4b-b722-cb41e0e1e1d8\") " pod="openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt"
Sep 06 19:28:23.630976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:23.630979    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6626becc-b3d3-4c4b-b722-cb41e0e1e1d8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt\" (UID: \"6626becc-b3d3-4c4b-b722-cb41e0e1e1d8\") " pod="openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt"
Sep 06 19:28:23.731883 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:23.731846    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgpc6\" (UniqueName: \"kubernetes.io/projected/6626becc-b3d3-4c4b-b722-cb41e0e1e1d8-kube-api-access-dgpc6\") pod \"payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt\" (UID: \"6626becc-b3d3-4c4b-b722-cb41e0e1e1d8\") " pod="openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt"
Sep 06 19:28:23.731883 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:23.731888    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6626becc-b3d3-4c4b-b722-cb41e0e1e1d8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt\" (UID: \"6626becc-b3d3-4c4b-b722-cb41e0e1e1d8\") " pod="openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt"
Sep 06 19:28:23.732566 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:23.732546    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6626becc-b3d3-4c4b-b722-cb41e0e1e1d8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt\" (UID: \"6626becc-b3d3-4c4b-b722-cb41e0e1e1d8\") " pod="openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt"
Sep 06 19:28:23.739677 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:23.739655    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgpc6\" (UniqueName: \"kubernetes.io/projected/6626becc-b3d3-4c4b-b722-cb41e0e1e1d8-kube-api-access-dgpc6\") pod \"payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt\" (UID: \"6626becc-b3d3-4c4b-b722-cb41e0e1e1d8\") " pod="openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt"
Sep 06 19:28:23.840088 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:23.840023    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt"
Sep 06 19:28:24.111301 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:24.111278    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt"]
Sep 06 19:28:24.112911 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:28:24.112883    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6626becc_b3d3_4c4b_b722_cb41e0e1e1d8.slice/crio-f352dab46c16733034a56d7158fe8c96640694d95d1ccf51be6001d2cbabf101 WatchSource:0}: Error finding container f352dab46c16733034a56d7158fe8c96640694d95d1ccf51be6001d2cbabf101: Status 404 returned error can't find the container with id f352dab46c16733034a56d7158fe8c96640694d95d1ccf51be6001d2cbabf101
Sep 06 19:28:24.953396 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:24.953348    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt" event={"ID":"6626becc-b3d3-4c4b-b722-cb41e0e1e1d8","Type":"ContainerStarted","Data":"5743ca3ccf59df14a17813f563837b965071246a6724908b2353f3466d9a52e2"}
Sep 06 19:28:24.953798 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:24.953408    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt" event={"ID":"6626becc-b3d3-4c4b-b722-cb41e0e1e1d8","Type":"ContainerStarted","Data":"f352dab46c16733034a56d7158fe8c96640694d95d1ccf51be6001d2cbabf101"}
Sep 06 19:28:24.953798 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:24.953434    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt"
Sep 06 19:28:24.953956 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:24.953937    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:28:24.971890 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:24.971831    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt" podStartSLOduration=1.971812124 podStartE2EDuration="1.971812124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:28:24.97068539 +0000 UTC m=+1039.151867198" watchObservedRunningTime="2026-09-06 19:28:24.971812124 +0000 UTC m=+1039.152993931"
Sep 06 19:28:35.960682 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:35.960652    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt"
Sep 06 19:28:39.748207 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:39.748171    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:28:50.027744 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.027704    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"]
Sep 06 19:28:50.032424 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.032403    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:28:50.034867 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.034834    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-4067c966-kserve-self-signed-certs\""
Sep 06 19:28:50.039118 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.039072    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"]
Sep 06 19:28:50.157120 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.157086    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-dshm\") pod \"e2e-conflict-b-4067c966-kserve-58f578857d-67hdh\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") " pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:28:50.157308 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.157126    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh572\" (UniqueName: \"kubernetes.io/projected/2e814b9c-be48-4eef-b6c2-370e7b20324c-kube-api-access-mh572\") pod \"e2e-conflict-b-4067c966-kserve-58f578857d-67hdh\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") " pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:28:50.157308 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.157195    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-model-cache\") pod \"e2e-conflict-b-4067c966-kserve-58f578857d-67hdh\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") " pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:28:50.157308 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.157221    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e814b9c-be48-4eef-b6c2-370e7b20324c-tls-certs\") pod \"e2e-conflict-b-4067c966-kserve-58f578857d-67hdh\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") " pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:28:50.157427 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.157310    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-home\") pod \"e2e-conflict-b-4067c966-kserve-58f578857d-67hdh\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") " pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:28:50.157427 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.157341    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-tmp-dir\") pod \"e2e-conflict-b-4067c966-kserve-58f578857d-67hdh\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") " pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:28:50.258825 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.258793    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e814b9c-be48-4eef-b6c2-370e7b20324c-tls-certs\") pod \"e2e-conflict-b-4067c966-kserve-58f578857d-67hdh\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") " pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:28:50.258825 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.258831    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-home\") pod \"e2e-conflict-b-4067c966-kserve-58f578857d-67hdh\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") " pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:28:50.259039 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.258848    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-tmp-dir\") pod \"e2e-conflict-b-4067c966-kserve-58f578857d-67hdh\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") " pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:28:50.259039 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.258912    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-dshm\") pod \"e2e-conflict-b-4067c966-kserve-58f578857d-67hdh\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") " pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:28:50.259039 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.258930    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh572\" (UniqueName: \"kubernetes.io/projected/2e814b9c-be48-4eef-b6c2-370e7b20324c-kube-api-access-mh572\") pod \"e2e-conflict-b-4067c966-kserve-58f578857d-67hdh\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") " pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:28:50.259039 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.258991    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-model-cache\") pod \"e2e-conflict-b-4067c966-kserve-58f578857d-67hdh\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") " pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:28:50.259252 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.259230    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-home\") pod \"e2e-conflict-b-4067c966-kserve-58f578857d-67hdh\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") " pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:28:50.259320 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.259286    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-tmp-dir\") pod \"e2e-conflict-b-4067c966-kserve-58f578857d-67hdh\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") " pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:28:50.259375 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.259320    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-model-cache\") pod \"e2e-conflict-b-4067c966-kserve-58f578857d-67hdh\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") " pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:28:50.261398 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.261377    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-dshm\") pod \"e2e-conflict-b-4067c966-kserve-58f578857d-67hdh\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") " pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:28:50.261515 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.261498    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e814b9c-be48-4eef-b6c2-370e7b20324c-tls-certs\") pod \"e2e-conflict-b-4067c966-kserve-58f578857d-67hdh\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") " pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:28:50.266655 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.266633    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh572\" (UniqueName: \"kubernetes.io/projected/2e814b9c-be48-4eef-b6c2-370e7b20324c-kube-api-access-mh572\") pod \"e2e-conflict-b-4067c966-kserve-58f578857d-67hdh\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") " pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:28:50.344181 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.344117    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:28:50.623330 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:50.623303    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"]
Sep 06 19:28:50.624367 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:28:50.624338    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e814b9c_be48_4eef_b6c2_370e7b20324c.slice/crio-f95d562f899228091c293b8b72d1e19f70b9c6a9947fb8304e5fb083cb1ddb9b WatchSource:0}: Error finding container f95d562f899228091c293b8b72d1e19f70b9c6a9947fb8304e5fb083cb1ddb9b: Status 404 returned error can't find the container with id f95d562f899228091c293b8b72d1e19f70b9c6a9947fb8304e5fb083cb1ddb9b
Sep 06 19:28:51.099918 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:51.099879    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh" event={"ID":"2e814b9c-be48-4eef-b6c2-370e7b20324c","Type":"ContainerStarted","Data":"7ebd855e02f7d81dd3491b85abc6157d04e65c5f763ea8c0ced201dad3894479"}
Sep 06 19:28:51.100386 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:51.099925    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh" event={"ID":"2e814b9c-be48-4eef-b6c2-370e7b20324c","Type":"ContainerStarted","Data":"f95d562f899228091c293b8b72d1e19f70b9c6a9947fb8304e5fb083cb1ddb9b"}
Sep 06 19:28:51.117717 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:51.117667    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh" podStartSLOduration=1.11765265 podStartE2EDuration="1.11765265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:28:51.114846382 +0000 UTC m=+1065.296028188" watchObservedRunningTime="2026-09-06 19:28:51.11765265 +0000 UTC m=+1065.298834456"
Sep 06 19:28:53.154141 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:53.154113    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:28:54.202050 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:54.202006    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt"]
Sep 06 19:28:54.202502 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:54.202285    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt" podUID="6626becc-b3d3-4c4b-b722-cb41e0e1e1d8" containerName="payload-processing" containerID="cri-o://5743ca3ccf59df14a17813f563837b965071246a6724908b2353f3466d9a52e2" gracePeriod=30
Sep 06 19:28:54.455302 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:54.455236    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt"
Sep 06 19:28:54.497512 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:54.497475    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6626becc-b3d3-4c4b-b722-cb41e0e1e1d8-plugins-config-volume\") pod \"6626becc-b3d3-4c4b-b722-cb41e0e1e1d8\" (UID: \"6626becc-b3d3-4c4b-b722-cb41e0e1e1d8\") "
Sep 06 19:28:54.497730 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:54.497567    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgpc6\" (UniqueName: \"kubernetes.io/projected/6626becc-b3d3-4c4b-b722-cb41e0e1e1d8-kube-api-access-dgpc6\") pod \"6626becc-b3d3-4c4b-b722-cb41e0e1e1d8\" (UID: \"6626becc-b3d3-4c4b-b722-cb41e0e1e1d8\") "
Sep 06 19:28:54.497841 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:54.497816    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6626becc-b3d3-4c4b-b722-cb41e0e1e1d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6626becc-b3d3-4c4b-b722-cb41e0e1e1d8" (UID: "6626becc-b3d3-4c4b-b722-cb41e0e1e1d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:28:54.497930 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:54.497915    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6626becc-b3d3-4c4b-b722-cb41e0e1e1d8-plugins-config-volume\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:28:54.499759 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:54.499731    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6626becc-b3d3-4c4b-b722-cb41e0e1e1d8-kube-api-access-dgpc6" (OuterVolumeSpecName: "kube-api-access-dgpc6") pod "6626becc-b3d3-4c4b-b722-cb41e0e1e1d8" (UID: "6626becc-b3d3-4c4b-b722-cb41e0e1e1d8"). InnerVolumeSpecName "kube-api-access-dgpc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:28:54.599343 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:54.599312    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgpc6\" (UniqueName: \"kubernetes.io/projected/6626becc-b3d3-4c4b-b722-cb41e0e1e1d8-kube-api-access-dgpc6\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:28:55.123451 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:55.123417    2566 generic.go:356] "Generic (PLEG): container finished" podID="6626becc-b3d3-4c4b-b722-cb41e0e1e1d8" containerID="5743ca3ccf59df14a17813f563837b965071246a6724908b2353f3466d9a52e2" exitCode=0
Sep 06 19:28:55.123648 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:55.123462    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt"
Sep 06 19:28:55.123648 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:55.123500    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt" event={"ID":"6626becc-b3d3-4c4b-b722-cb41e0e1e1d8","Type":"ContainerDied","Data":"5743ca3ccf59df14a17813f563837b965071246a6724908b2353f3466d9a52e2"}
Sep 06 19:28:55.123648 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:55.123557    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt" event={"ID":"6626becc-b3d3-4c4b-b722-cb41e0e1e1d8","Type":"ContainerDied","Data":"f352dab46c16733034a56d7158fe8c96640694d95d1ccf51be6001d2cbabf101"}
Sep 06 19:28:55.123648 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:55.123586    2566 scope.go:117] "RemoveContainer" containerID="5743ca3ccf59df14a17813f563837b965071246a6724908b2353f3466d9a52e2"
Sep 06 19:28:55.133551 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:55.133536    2566 scope.go:117] "RemoveContainer" containerID="5743ca3ccf59df14a17813f563837b965071246a6724908b2353f3466d9a52e2"
Sep 06 19:28:55.133825 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:28:55.133808    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5743ca3ccf59df14a17813f563837b965071246a6724908b2353f3466d9a52e2\": container with ID starting with 5743ca3ccf59df14a17813f563837b965071246a6724908b2353f3466d9a52e2 not found: ID does not exist" containerID="5743ca3ccf59df14a17813f563837b965071246a6724908b2353f3466d9a52e2"
Sep 06 19:28:55.133899 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:55.133831    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5743ca3ccf59df14a17813f563837b965071246a6724908b2353f3466d9a52e2"} err="failed to get container status \"5743ca3ccf59df14a17813f563837b965071246a6724908b2353f3466d9a52e2\": rpc error: code = NotFound desc = could not find container \"5743ca3ccf59df14a17813f563837b965071246a6724908b2353f3466d9a52e2\": container with ID starting with 5743ca3ccf59df14a17813f563837b965071246a6724908b2353f3466d9a52e2 not found: ID does not exist"
Sep 06 19:28:55.147424 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:55.147390    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt"]
Sep 06 19:28:55.154206 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:55.154183    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-053eb1b8-7c5668bf4b-wmxpt"]
Sep 06 19:28:56.310685 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:28:56.310657    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6626becc-b3d3-4c4b-b722-cb41e0e1e1d8" path="/var/lib/kubelet/pods/6626becc-b3d3-4c4b-b722-cb41e0e1e1d8/volumes"
Sep 06 19:29:00.345124 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:29:00.345047    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:29:00.345124 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:29:00.345087    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:29:00.357450 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:29:00.357431    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:29:01.174740 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:29:01.174713    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:29:12.445562 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:29:12.445522    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:29:27.041690 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:29:27.041646    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:29:44.753614 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:29:44.753559    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:30:00.132386 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:00.132345    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5307a7-29812050-sh24g"]
Sep 06 19:30:00.132778 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:00.132769    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6626becc-b3d3-4c4b-b722-cb41e0e1e1d8" containerName="payload-processing"
Sep 06 19:30:00.132821 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:00.132781    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6626becc-b3d3-4c4b-b722-cb41e0e1e1d8" containerName="payload-processing"
Sep 06 19:30:00.132860 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:00.132851    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6626becc-b3d3-4c4b-b722-cb41e0e1e1d8" containerName="payload-processing"
Sep 06 19:30:00.136110 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:00.136089    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5307a7-29812050-sh24g"
Sep 06 19:30:00.138362 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:00.138335    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d73678-29812050-mp227"]
Sep 06 19:30:00.138599 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:00.138580    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpc8j\""
Sep 06 19:30:00.142354 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:00.142325    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d73678-29812050-mp227"
Sep 06 19:30:00.145536 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:00.145514    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5307a7-29812050-sh24g"]
Sep 06 19:30:00.149012 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:00.148986    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d73678-29812050-mp227"]
Sep 06 19:30:00.150469 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:00.150448    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5307a7-29812050-sh24g"
Sep 06 19:30:00.158356 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:00.158336    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d73678-29812050-mp227"
Sep 06 19:30:00.429452 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:00.429415    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5307a7-29812050-sh24g"]
Sep 06 19:30:00.431268 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:30:00.431240    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be75dec_2b86_41b5_a254_a83265cd08e6.slice/crio-cb7d9f608e97e801ace917a8cfd8cf9743c6239f85939b123e489ba83d304f38 WatchSource:0}: Error finding container cb7d9f608e97e801ace917a8cfd8cf9743c6239f85939b123e489ba83d304f38: Status 404 returned error can't find the container with id cb7d9f608e97e801ace917a8cfd8cf9743c6239f85939b123e489ba83d304f38
Sep 06 19:30:00.438051 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:00.438032    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d73678-29812050-mp227"]
Sep 06 19:30:00.439948 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:30:00.439925    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa44ab2_ee5e_4e29_b873_e908fd6de7f3.slice/crio-71b4f70dbbf98d97e38062ff68df45366c22de07b0474d875d8df16653532b02 WatchSource:0}: Error finding container 71b4f70dbbf98d97e38062ff68df45366c22de07b0474d875d8df16653532b02: Status 404 returned error can't find the container with id 71b4f70dbbf98d97e38062ff68df45366c22de07b0474d875d8df16653532b02
Sep 06 19:30:00.499669 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:00.499634    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5307a7-29812050-sh24g" event={"ID":"8be75dec-2b86-41b5-a254-a83265cd08e6","Type":"ContainerStarted","Data":"cb7d9f608e97e801ace917a8cfd8cf9743c6239f85939b123e489ba83d304f38"}
Sep 06 19:30:00.500944 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:00.500924    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d73678-29812050-mp227" event={"ID":"aaa44ab2-ee5e-4e29-b873-e908fd6de7f3","Type":"ContainerStarted","Data":"71b4f70dbbf98d97e38062ff68df45366c22de07b0474d875d8df16653532b02"}
Sep 06 19:30:03.523184 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:03.523147    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5307a7-29812050-sh24g" event={"ID":"8be75dec-2b86-41b5-a254-a83265cd08e6","Type":"ContainerStarted","Data":"5cdcea514930b52f6fdeaa261834b520f4a348ae81922badda21afedbef2930a"}
Sep 06 19:30:03.524968 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:03.524938    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d73678-29812050-mp227" event={"ID":"aaa44ab2-ee5e-4e29-b873-e908fd6de7f3","Type":"ContainerStarted","Data":"1fe557b7016e654ec7dbde1455783958cb8113f62bc88266860ffe6db74a1b6b"}
Sep 06 19:30:03.539561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:03.539521    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5307a7-29812050-sh24g" podStartSLOduration=1.330246444 podStartE2EDuration="3.539506737s" totalImagesPullingTime="2.209260293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:30:03.536569387 +0000 UTC m=+1137.717751195" watchObservedRunningTime="2026-09-06 19:30:03.539506737 +0000 UTC m=+1137.720688543"
Sep 06 19:30:03.554988 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:03.554948    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d73678-29812050-mp227" podStartSLOduration=1.358338923 podStartE2EDuration="3.554934833s" totalImagesPullingTime="2.19659591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:30:03.55362914 +0000 UTC m=+1137.734810948" watchObservedRunningTime="2026-09-06 19:30:03.554934833 +0000 UTC m=+1137.736116638"
Sep 06 19:30:04.531740 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:04.531705    2566 generic.go:356] "Generic (PLEG): container finished" podID="8be75dec-2b86-41b5-a254-a83265cd08e6" containerID="5cdcea514930b52f6fdeaa261834b520f4a348ae81922badda21afedbef2930a" exitCode=0
Sep 06 19:30:04.532131 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:04.531787    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5307a7-29812050-sh24g" event={"ID":"8be75dec-2b86-41b5-a254-a83265cd08e6","Type":"ContainerDied","Data":"5cdcea514930b52f6fdeaa261834b520f4a348ae81922badda21afedbef2930a"}
Sep 06 19:30:04.533332 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:04.533309    2566 generic.go:356] "Generic (PLEG): container finished" podID="aaa44ab2-ee5e-4e29-b873-e908fd6de7f3" containerID="1fe557b7016e654ec7dbde1455783958cb8113f62bc88266860ffe6db74a1b6b" exitCode=0
Sep 06 19:30:04.533421 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:04.533396    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d73678-29812050-mp227" event={"ID":"aaa44ab2-ee5e-4e29-b873-e908fd6de7f3","Type":"ContainerDied","Data":"1fe557b7016e654ec7dbde1455783958cb8113f62bc88266860ffe6db74a1b6b"}
Sep 06 19:30:05.664528 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:05.664505    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5307a7-29812050-sh24g"
Sep 06 19:30:05.706923 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:05.706898    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d73678-29812050-mp227"
Sep 06 19:30:06.545512 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:06.545486    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5307a7-29812050-sh24g"
Sep 06 19:30:06.545512 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:06.545501    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5307a7-29812050-sh24g" event={"ID":"8be75dec-2b86-41b5-a254-a83265cd08e6","Type":"ContainerDied","Data":"cb7d9f608e97e801ace917a8cfd8cf9743c6239f85939b123e489ba83d304f38"}
Sep 06 19:30:06.545774 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:06.545575    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb7d9f608e97e801ace917a8cfd8cf9743c6239f85939b123e489ba83d304f38"
Sep 06 19:30:06.546910 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:06.546886    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d73678-29812050-mp227" event={"ID":"aaa44ab2-ee5e-4e29-b873-e908fd6de7f3","Type":"ContainerDied","Data":"71b4f70dbbf98d97e38062ff68df45366c22de07b0474d875d8df16653532b02"}
Sep 06 19:30:06.547013 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:06.546928    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71b4f70dbbf98d97e38062ff68df45366c22de07b0474d875d8df16653532b02"
Sep 06 19:30:06.547013 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:06.546894    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d73678-29812050-mp227"
Sep 06 19:30:08.840110 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:08.840080    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"]
Sep 06 19:30:08.840484 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:08.840456    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be75dec-2b86-41b5-a254-a83265cd08e6" containerName="cleanup"
Sep 06 19:30:08.840484 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:08.840467    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be75dec-2b86-41b5-a254-a83265cd08e6" containerName="cleanup"
Sep 06 19:30:08.840484 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:08.840476    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa44ab2-ee5e-4e29-b873-e908fd6de7f3" containerName="cleanup"
Sep 06 19:30:08.840484 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:08.840481    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa44ab2-ee5e-4e29-b873-e908fd6de7f3" containerName="cleanup"
Sep 06 19:30:08.840655 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:08.840549    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaa44ab2-ee5e-4e29-b873-e908fd6de7f3" containerName="cleanup"
Sep 06 19:30:08.840655 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:08.840558    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8be75dec-2b86-41b5-a254-a83265cd08e6" containerName="cleanup"
Sep 06 19:30:08.843715 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:08.843688    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:08.847248 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:08.847224    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-21b01e\"/\"default-dockercfg-64cjp\""
Sep 06 19:30:08.847248 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:08.847241    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-21b01e\"/\"kube-root-ca.crt\""
Sep 06 19:30:08.847442 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:08.847235    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-21b01e\"/\"autores-model-21b01e-kserve-self-signed-certs\""
Sep 06 19:30:08.847442 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:08.847262    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-21b01e\"/\"openshift-service-ca.crt\""
Sep 06 19:30:08.852183 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:08.852151    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"]
Sep 06 19:30:08.947755 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:08.947722    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:30:08.953506 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:08.953484    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-dshm\") pod \"autores-model-21b01e-kserve-5cb59589c9-7kq8q\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") " pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:08.953627 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:08.953518    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7c913e-65e8-441b-b40d-17216ac793d8-tls-certs\") pod \"autores-model-21b01e-kserve-5cb59589c9-7kq8q\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") " pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:08.953627 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:08.953541    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxgl6\" (UniqueName: \"kubernetes.io/projected/2b7c913e-65e8-441b-b40d-17216ac793d8-kube-api-access-kxgl6\") pod \"autores-model-21b01e-kserve-5cb59589c9-7kq8q\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") " pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:08.953627 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:08.953582    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-home\") pod \"autores-model-21b01e-kserve-5cb59589c9-7kq8q\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") " pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:08.953799 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:08.953637    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-tmp-dir\") pod \"autores-model-21b01e-kserve-5cb59589c9-7kq8q\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") " pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:08.953799 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:08.953729    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-model-cache\") pod \"autores-model-21b01e-kserve-5cb59589c9-7kq8q\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") " pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:09.054487 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:09.054453    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-model-cache\") pod \"autores-model-21b01e-kserve-5cb59589c9-7kq8q\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") " pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:09.054666 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:09.054645    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-dshm\") pod \"autores-model-21b01e-kserve-5cb59589c9-7kq8q\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") " pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:09.054722 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:09.054698    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7c913e-65e8-441b-b40d-17216ac793d8-tls-certs\") pod \"autores-model-21b01e-kserve-5cb59589c9-7kq8q\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") " pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:09.054762 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:09.054741    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxgl6\" (UniqueName: \"kubernetes.io/projected/2b7c913e-65e8-441b-b40d-17216ac793d8-kube-api-access-kxgl6\") pod \"autores-model-21b01e-kserve-5cb59589c9-7kq8q\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") " pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:09.054815 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:09.054781    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-home\") pod \"autores-model-21b01e-kserve-5cb59589c9-7kq8q\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") " pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:09.054869 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:09.054833    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-tmp-dir\") pod \"autores-model-21b01e-kserve-5cb59589c9-7kq8q\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") " pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:09.054920 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:09.054892    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-model-cache\") pod \"autores-model-21b01e-kserve-5cb59589c9-7kq8q\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") " pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:09.055159 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:09.055133    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-tmp-dir\") pod \"autores-model-21b01e-kserve-5cb59589c9-7kq8q\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") " pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:09.055159 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:09.055155    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-home\") pod \"autores-model-21b01e-kserve-5cb59589c9-7kq8q\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") " pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:09.057026 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:09.057002    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-dshm\") pod \"autores-model-21b01e-kserve-5cb59589c9-7kq8q\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") " pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:09.057279 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:09.057265    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7c913e-65e8-441b-b40d-17216ac793d8-tls-certs\") pod \"autores-model-21b01e-kserve-5cb59589c9-7kq8q\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") " pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:09.061942 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:09.061922    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxgl6\" (UniqueName: \"kubernetes.io/projected/2b7c913e-65e8-441b-b40d-17216ac793d8-kube-api-access-kxgl6\") pod \"autores-model-21b01e-kserve-5cb59589c9-7kq8q\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") " pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:09.154240 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:09.154157    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:09.428075 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:09.428047    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"]
Sep 06 19:30:09.429669 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:30:09.429643    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b7c913e_65e8_441b_b40d_17216ac793d8.slice/crio-07f65e25b1cbb24b95d1569a1ff67b7e47b3532273092823ae7e9ada8e118865 WatchSource:0}: Error finding container 07f65e25b1cbb24b95d1569a1ff67b7e47b3532273092823ae7e9ada8e118865: Status 404 returned error can't find the container with id 07f65e25b1cbb24b95d1569a1ff67b7e47b3532273092823ae7e9ada8e118865
Sep 06 19:30:09.565470 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:09.565435    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q" event={"ID":"2b7c913e-65e8-441b-b40d-17216ac793d8","Type":"ContainerStarted","Data":"dd645316676ec92ec604d3968f9871e429d2ec8ea0808f898d0f5562f256a855"}
Sep 06 19:30:09.565640 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:09.565484    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q" event={"ID":"2b7c913e-65e8-441b-b40d-17216ac793d8","Type":"ContainerStarted","Data":"07f65e25b1cbb24b95d1569a1ff67b7e47b3532273092823ae7e9ada8e118865"}
Sep 06 19:30:09.581694 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:09.581649    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q" podStartSLOduration=1.581634548 podStartE2EDuration="1.581634548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:30:09.580083743 +0000 UTC m=+1143.761265549" watchObservedRunningTime="2026-09-06 19:30:09.581634548 +0000 UTC m=+1143.762816350"
Sep 06 19:30:19.154407 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:19.154361    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:19.154907 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:19.154434    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:19.166808 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:19.166789    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:19.634233 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:19.634205    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:23.243209 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:23.243177    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:30:30.150360 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:30.150288    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"]
Sep 06 19:30:30.631421 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:30.631391    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"]
Sep 06 19:30:34.055116 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:34.055075    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:30:41.270073 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.270039    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp"]
Sep 06 19:30:41.270542 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.270317    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp" podUID="29ce5aa5-2753-429d-bfb0-1493556b5999" containerName="payload-processing" containerID="cri-o://a95b50a9972880f825016b7a267a976bdf34f2155b4dc085f4fd64f59ee7db89" gracePeriod=30
Sep 06 19:30:41.276125 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.276098    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff"]
Sep 06 19:30:41.276528 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.276484    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff" podUID="b631e3c5-9c83-47ea-88d7-b02208086dcf" containerName="payload-pre-processing" containerID="cri-o://82f65287ce141b97a73b8c9597a1f90db6e81739d73ae8e7945db28f5fe2a7a0" gracePeriod=30
Sep 06 19:30:41.547563 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.547537    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp"
Sep 06 19:30:41.550663 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.550637    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff"
Sep 06 19:30:41.654201 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.654168    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b631e3c5-9c83-47ea-88d7-b02208086dcf-plugins-config-volume\") pod \"b631e3c5-9c83-47ea-88d7-b02208086dcf\" (UID: \"b631e3c5-9c83-47ea-88d7-b02208086dcf\") "
Sep 06 19:30:41.654364 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.654288    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7fxr\" (UniqueName: \"kubernetes.io/projected/29ce5aa5-2753-429d-bfb0-1493556b5999-kube-api-access-q7fxr\") pod \"29ce5aa5-2753-429d-bfb0-1493556b5999\" (UID: \"29ce5aa5-2753-429d-bfb0-1493556b5999\") "
Sep 06 19:30:41.654364 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.654322    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g27xh\" (UniqueName: \"kubernetes.io/projected/b631e3c5-9c83-47ea-88d7-b02208086dcf-kube-api-access-g27xh\") pod \"b631e3c5-9c83-47ea-88d7-b02208086dcf\" (UID: \"b631e3c5-9c83-47ea-88d7-b02208086dcf\") "
Sep 06 19:30:41.654503 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.654394    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ce5aa5-2753-429d-bfb0-1493556b5999-plugins-config-volume\") pod \"29ce5aa5-2753-429d-bfb0-1493556b5999\" (UID: \"29ce5aa5-2753-429d-bfb0-1493556b5999\") "
Sep 06 19:30:41.654555 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.654515    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b631e3c5-9c83-47ea-88d7-b02208086dcf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b631e3c5-9c83-47ea-88d7-b02208086dcf" (UID: "b631e3c5-9c83-47ea-88d7-b02208086dcf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:30:41.654830 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.654805    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29ce5aa5-2753-429d-bfb0-1493556b5999-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "29ce5aa5-2753-429d-bfb0-1493556b5999" (UID: "29ce5aa5-2753-429d-bfb0-1493556b5999"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:30:41.654914 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.654848    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b631e3c5-9c83-47ea-88d7-b02208086dcf-plugins-config-volume\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:30:41.656484 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.656459    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b631e3c5-9c83-47ea-88d7-b02208086dcf-kube-api-access-g27xh" (OuterVolumeSpecName: "kube-api-access-g27xh") pod "b631e3c5-9c83-47ea-88d7-b02208086dcf" (UID: "b631e3c5-9c83-47ea-88d7-b02208086dcf"). InnerVolumeSpecName "kube-api-access-g27xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:30:41.656580 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.656484    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ce5aa5-2753-429d-bfb0-1493556b5999-kube-api-access-q7fxr" (OuterVolumeSpecName: "kube-api-access-q7fxr") pod "29ce5aa5-2753-429d-bfb0-1493556b5999" (UID: "29ce5aa5-2753-429d-bfb0-1493556b5999"). InnerVolumeSpecName "kube-api-access-q7fxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:30:41.746985 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.746955    2566 generic.go:356] "Generic (PLEG): container finished" podID="29ce5aa5-2753-429d-bfb0-1493556b5999" containerID="a95b50a9972880f825016b7a267a976bdf34f2155b4dc085f4fd64f59ee7db89" exitCode=0
Sep 06 19:30:41.747130 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.746998    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp"
Sep 06 19:30:41.747130 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.747041    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp" event={"ID":"29ce5aa5-2753-429d-bfb0-1493556b5999","Type":"ContainerDied","Data":"a95b50a9972880f825016b7a267a976bdf34f2155b4dc085f4fd64f59ee7db89"}
Sep 06 19:30:41.747130 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.747093    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp" event={"ID":"29ce5aa5-2753-429d-bfb0-1493556b5999","Type":"ContainerDied","Data":"3fcbe48f8308c1aa0aac84dcfb46c5b4a800523f583ff76e8a8add8952769d0a"}
Sep 06 19:30:41.747130 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.747120    2566 scope.go:117] "RemoveContainer" containerID="a95b50a9972880f825016b7a267a976bdf34f2155b4dc085f4fd64f59ee7db89"
Sep 06 19:30:41.748475 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.748457    2566 generic.go:356] "Generic (PLEG): container finished" podID="b631e3c5-9c83-47ea-88d7-b02208086dcf" containerID="82f65287ce141b97a73b8c9597a1f90db6e81739d73ae8e7945db28f5fe2a7a0" exitCode=0
Sep 06 19:30:41.748564 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.748499    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff" event={"ID":"b631e3c5-9c83-47ea-88d7-b02208086dcf","Type":"ContainerDied","Data":"82f65287ce141b97a73b8c9597a1f90db6e81739d73ae8e7945db28f5fe2a7a0"}
Sep 06 19:30:41.748564 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.748509    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff"
Sep 06 19:30:41.748564 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.748533    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff" event={"ID":"b631e3c5-9c83-47ea-88d7-b02208086dcf","Type":"ContainerDied","Data":"7932a14af776395e10732fb3334ae57c76ddd0bda393e831a87c16f52821469a"}
Sep 06 19:30:41.755222 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.755200    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ce5aa5-2753-429d-bfb0-1493556b5999-plugins-config-volume\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:30:41.755222 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.755223    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7fxr\" (UniqueName: \"kubernetes.io/projected/29ce5aa5-2753-429d-bfb0-1493556b5999-kube-api-access-q7fxr\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:30:41.755352 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.755233    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g27xh\" (UniqueName: \"kubernetes.io/projected/b631e3c5-9c83-47ea-88d7-b02208086dcf-kube-api-access-g27xh\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:30:41.757060 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.757040    2566 scope.go:117] "RemoveContainer" containerID="a95b50a9972880f825016b7a267a976bdf34f2155b4dc085f4fd64f59ee7db89"
Sep 06 19:30:41.757350 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:30:41.757326    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a95b50a9972880f825016b7a267a976bdf34f2155b4dc085f4fd64f59ee7db89\": container with ID starting with a95b50a9972880f825016b7a267a976bdf34f2155b4dc085f4fd64f59ee7db89 not found: ID does not exist" containerID="a95b50a9972880f825016b7a267a976bdf34f2155b4dc085f4fd64f59ee7db89"
Sep 06 19:30:41.757410 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.757364    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a95b50a9972880f825016b7a267a976bdf34f2155b4dc085f4fd64f59ee7db89"} err="failed to get container status \"a95b50a9972880f825016b7a267a976bdf34f2155b4dc085f4fd64f59ee7db89\": rpc error: code = NotFound desc = could not find container \"a95b50a9972880f825016b7a267a976bdf34f2155b4dc085f4fd64f59ee7db89\": container with ID starting with a95b50a9972880f825016b7a267a976bdf34f2155b4dc085f4fd64f59ee7db89 not found: ID does not exist"
Sep 06 19:30:41.757410 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.757388    2566 scope.go:117] "RemoveContainer" containerID="82f65287ce141b97a73b8c9597a1f90db6e81739d73ae8e7945db28f5fe2a7a0"
Sep 06 19:30:41.766261 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.766247    2566 scope.go:117] "RemoveContainer" containerID="82f65287ce141b97a73b8c9597a1f90db6e81739d73ae8e7945db28f5fe2a7a0"
Sep 06 19:30:41.766510 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:30:41.766490    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f65287ce141b97a73b8c9597a1f90db6e81739d73ae8e7945db28f5fe2a7a0\": container with ID starting with 82f65287ce141b97a73b8c9597a1f90db6e81739d73ae8e7945db28f5fe2a7a0 not found: ID does not exist" containerID="82f65287ce141b97a73b8c9597a1f90db6e81739d73ae8e7945db28f5fe2a7a0"
Sep 06 19:30:41.766571 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.766517    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f65287ce141b97a73b8c9597a1f90db6e81739d73ae8e7945db28f5fe2a7a0"} err="failed to get container status \"82f65287ce141b97a73b8c9597a1f90db6e81739d73ae8e7945db28f5fe2a7a0\": rpc error: code = NotFound desc = could not find container \"82f65287ce141b97a73b8c9597a1f90db6e81739d73ae8e7945db28f5fe2a7a0\": container with ID starting with 82f65287ce141b97a73b8c9597a1f90db6e81739d73ae8e7945db28f5fe2a7a0 not found: ID does not exist"
Sep 06 19:30:41.772807 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.772761    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff"]
Sep 06 19:30:41.776171 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.776149    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-21b01e-7fb59b8f78-p9vff"]
Sep 06 19:30:41.786264 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.786244    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp"]
Sep 06 19:30:41.788491 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:41.788475    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-21b01e-6674886df9-dd6kp"]
Sep 06 19:30:42.311227 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:42.311197    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29ce5aa5-2753-429d-bfb0-1493556b5999" path="/var/lib/kubelet/pods/29ce5aa5-2753-429d-bfb0-1493556b5999/volumes"
Sep 06 19:30:42.311578 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:42.311533    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b631e3c5-9c83-47ea-88d7-b02208086dcf" path="/var/lib/kubelet/pods/b631e3c5-9c83-47ea-88d7-b02208086dcf/volumes"
Sep 06 19:30:45.170740 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.170698    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh" podUID="2e814b9c-be48-4eef-b6c2-370e7b20324c" containerName="main" containerID="cri-o://7ebd855e02f7d81dd3491b85abc6157d04e65c5f763ea8c0ced201dad3894479" gracePeriod=45
Sep 06 19:30:45.424805 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.424745    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:30:45.488884 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.488854    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-dshm\") pod \"2e814b9c-be48-4eef-b6c2-370e7b20324c\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") "
Sep 06 19:30:45.489014 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.488892    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-model-cache\") pod \"2e814b9c-be48-4eef-b6c2-370e7b20324c\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") "
Sep 06 19:30:45.489014 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.488914    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-home\") pod \"2e814b9c-be48-4eef-b6c2-370e7b20324c\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") "
Sep 06 19:30:45.489014 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.488972    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e814b9c-be48-4eef-b6c2-370e7b20324c-tls-certs\") pod \"2e814b9c-be48-4eef-b6c2-370e7b20324c\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") "
Sep 06 19:30:45.489132 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.489042    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mh572\" (UniqueName: \"kubernetes.io/projected/2e814b9c-be48-4eef-b6c2-370e7b20324c-kube-api-access-mh572\") pod \"2e814b9c-be48-4eef-b6c2-370e7b20324c\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") "
Sep 06 19:30:45.489132 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.489090    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-tmp-dir\") pod \"2e814b9c-be48-4eef-b6c2-370e7b20324c\" (UID: \"2e814b9c-be48-4eef-b6c2-370e7b20324c\") "
Sep 06 19:30:45.489210 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.489180    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-home" (OuterVolumeSpecName: "home") pod "2e814b9c-be48-4eef-b6c2-370e7b20324c" (UID: "2e814b9c-be48-4eef-b6c2-370e7b20324c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:30:45.489210 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.489195    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e814b9c-be48-4eef-b6c2-370e7b20324c" (UID: "2e814b9c-be48-4eef-b6c2-370e7b20324c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:30:45.489381 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.489344    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e814b9c-be48-4eef-b6c2-370e7b20324c" (UID: "2e814b9c-be48-4eef-b6c2-370e7b20324c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:30:45.489439 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.489383    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-model-cache\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:30:45.489439 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.489405    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-home\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:30:45.491220 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.491196    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e814b9c-be48-4eef-b6c2-370e7b20324c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e814b9c-be48-4eef-b6c2-370e7b20324c" (UID: "2e814b9c-be48-4eef-b6c2-370e7b20324c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:30:45.491338 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.491257    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e814b9c-be48-4eef-b6c2-370e7b20324c-kube-api-access-mh572" (OuterVolumeSpecName: "kube-api-access-mh572") pod "2e814b9c-be48-4eef-b6c2-370e7b20324c" (UID: "2e814b9c-be48-4eef-b6c2-370e7b20324c"). InnerVolumeSpecName "kube-api-access-mh572". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:30:45.491338 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.491265    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-dshm" (OuterVolumeSpecName: "dshm") pod "2e814b9c-be48-4eef-b6c2-370e7b20324c" (UID: "2e814b9c-be48-4eef-b6c2-370e7b20324c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:30:45.590729 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.590697    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-dshm\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:30:45.590729 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.590722    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e814b9c-be48-4eef-b6c2-370e7b20324c-tls-certs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:30:45.590887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.590737    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh572\" (UniqueName: \"kubernetes.io/projected/2e814b9c-be48-4eef-b6c2-370e7b20324c-kube-api-access-mh572\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:30:45.590887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.590749    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e814b9c-be48-4eef-b6c2-370e7b20324c-tmp-dir\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:30:45.653575 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.653535    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q" podUID="2b7c913e-65e8-441b-b40d-17216ac793d8" containerName="main" containerID="cri-o://dd645316676ec92ec604d3968f9871e429d2ec8ea0808f898d0f5562f256a855" gracePeriod=45
Sep 06 19:30:45.774145 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.774114    2566 generic.go:356] "Generic (PLEG): container finished" podID="2e814b9c-be48-4eef-b6c2-370e7b20324c" containerID="7ebd855e02f7d81dd3491b85abc6157d04e65c5f763ea8c0ced201dad3894479" exitCode=0
Sep 06 19:30:45.774257 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.774160    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"
Sep 06 19:30:45.774257 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.774192    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh" event={"ID":"2e814b9c-be48-4eef-b6c2-370e7b20324c","Type":"ContainerDied","Data":"7ebd855e02f7d81dd3491b85abc6157d04e65c5f763ea8c0ced201dad3894479"}
Sep 06 19:30:45.774257 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.774246    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh" event={"ID":"2e814b9c-be48-4eef-b6c2-370e7b20324c","Type":"ContainerDied","Data":"f95d562f899228091c293b8b72d1e19f70b9c6a9947fb8304e5fb083cb1ddb9b"}
Sep 06 19:30:45.774422 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.774271    2566 scope.go:117] "RemoveContainer" containerID="7ebd855e02f7d81dd3491b85abc6157d04e65c5f763ea8c0ced201dad3894479"
Sep 06 19:30:45.776493 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.776456    2566 generic.go:356] "Generic (PLEG): container finished" podID="2b7c913e-65e8-441b-b40d-17216ac793d8" containerID="dd645316676ec92ec604d3968f9871e429d2ec8ea0808f898d0f5562f256a855" exitCode=0
Sep 06 19:30:45.776653 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.776491    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q" event={"ID":"2b7c913e-65e8-441b-b40d-17216ac793d8","Type":"ContainerDied","Data":"dd645316676ec92ec604d3968f9871e429d2ec8ea0808f898d0f5562f256a855"}
Sep 06 19:30:45.784494 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.784473    2566 scope.go:117] "RemoveContainer" containerID="7ebd855e02f7d81dd3491b85abc6157d04e65c5f763ea8c0ced201dad3894479"
Sep 06 19:30:45.784862 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:30:45.784838    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ebd855e02f7d81dd3491b85abc6157d04e65c5f763ea8c0ced201dad3894479\": container with ID starting with 7ebd855e02f7d81dd3491b85abc6157d04e65c5f763ea8c0ced201dad3894479 not found: ID does not exist" containerID="7ebd855e02f7d81dd3491b85abc6157d04e65c5f763ea8c0ced201dad3894479"
Sep 06 19:30:45.784962 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.784872    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ebd855e02f7d81dd3491b85abc6157d04e65c5f763ea8c0ced201dad3894479"} err="failed to get container status \"7ebd855e02f7d81dd3491b85abc6157d04e65c5f763ea8c0ced201dad3894479\": rpc error: code = NotFound desc = could not find container \"7ebd855e02f7d81dd3491b85abc6157d04e65c5f763ea8c0ced201dad3894479\": container with ID starting with 7ebd855e02f7d81dd3491b85abc6157d04e65c5f763ea8c0ced201dad3894479 not found: ID does not exist"
Sep 06 19:30:45.796022 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.795993    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"]
Sep 06 19:30:45.801469 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.801443    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-4067c966-kserve-58f578857d-67hdh"]
Sep 06 19:30:45.898418 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.898391    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:45.994758 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.994733    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-home\") pod \"2b7c913e-65e8-441b-b40d-17216ac793d8\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") "
Sep 06 19:30:45.994918 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.994764    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7c913e-65e8-441b-b40d-17216ac793d8-tls-certs\") pod \"2b7c913e-65e8-441b-b40d-17216ac793d8\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") "
Sep 06 19:30:45.994918 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.994805    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxgl6\" (UniqueName: \"kubernetes.io/projected/2b7c913e-65e8-441b-b40d-17216ac793d8-kube-api-access-kxgl6\") pod \"2b7c913e-65e8-441b-b40d-17216ac793d8\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") "
Sep 06 19:30:45.994918 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.994821    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-tmp-dir\") pod \"2b7c913e-65e8-441b-b40d-17216ac793d8\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") "
Sep 06 19:30:45.995429 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.995189    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-home" (OuterVolumeSpecName: "home") pod "2b7c913e-65e8-441b-b40d-17216ac793d8" (UID: "2b7c913e-65e8-441b-b40d-17216ac793d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:30:45.995429 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.995292    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2b7c913e-65e8-441b-b40d-17216ac793d8" (UID: "2b7c913e-65e8-441b-b40d-17216ac793d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:30:45.995429 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.995316    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-model-cache\") pod \"2b7c913e-65e8-441b-b40d-17216ac793d8\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") "
Sep 06 19:30:45.995429 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.995399    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-dshm\") pod \"2b7c913e-65e8-441b-b40d-17216ac793d8\" (UID: \"2b7c913e-65e8-441b-b40d-17216ac793d8\") "
Sep 06 19:30:45.995714 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.995582    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "2b7c913e-65e8-441b-b40d-17216ac793d8" (UID: "2b7c913e-65e8-441b-b40d-17216ac793d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:30:45.997562 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.996182    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-tmp-dir\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:30:45.997562 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.996212    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-model-cache\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:30:45.997562 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.996227    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-home\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:30:45.999650 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.998331    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b7c913e-65e8-441b-b40d-17216ac793d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b7c913e-65e8-441b-b40d-17216ac793d8" (UID: "2b7c913e-65e8-441b-b40d-17216ac793d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:30:45.999650 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.998392    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-dshm" (OuterVolumeSpecName: "dshm") pod "2b7c913e-65e8-441b-b40d-17216ac793d8" (UID: "2b7c913e-65e8-441b-b40d-17216ac793d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:30:45.999650 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:45.998415    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7c913e-65e8-441b-b40d-17216ac793d8-kube-api-access-kxgl6" (OuterVolumeSpecName: "kube-api-access-kxgl6") pod "2b7c913e-65e8-441b-b40d-17216ac793d8" (UID: "2b7c913e-65e8-441b-b40d-17216ac793d8"). InnerVolumeSpecName "kube-api-access-kxgl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:30:46.097517 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:46.097436    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxgl6\" (UniqueName: \"kubernetes.io/projected/2b7c913e-65e8-441b-b40d-17216ac793d8-kube-api-access-kxgl6\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:30:46.097517 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:46.097472    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b7c913e-65e8-441b-b40d-17216ac793d8-dshm\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:30:46.097517 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:46.097485    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7c913e-65e8-441b-b40d-17216ac793d8-tls-certs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:30:46.312210 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:46.312177    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e814b9c-be48-4eef-b6c2-370e7b20324c" path="/var/lib/kubelet/pods/2e814b9c-be48-4eef-b6c2-370e7b20324c/volumes"
Sep 06 19:30:46.785054 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:46.785016    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q" event={"ID":"2b7c913e-65e8-441b-b40d-17216ac793d8","Type":"ContainerDied","Data":"07f65e25b1cbb24b95d1569a1ff67b7e47b3532273092823ae7e9ada8e118865"}
Sep 06 19:30:46.785054 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:46.785042    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"
Sep 06 19:30:46.785238 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:46.785062    2566 scope.go:117] "RemoveContainer" containerID="dd645316676ec92ec604d3968f9871e429d2ec8ea0808f898d0f5562f256a855"
Sep 06 19:30:46.801221 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:46.801198    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"]
Sep 06 19:30:46.809873 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:46.809853    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-21b01e/autores-model-21b01e-kserve-5cb59589c9-7kq8q"]
Sep 06 19:30:47.400887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:47.400855    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:30:48.310134 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:48.310102    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b7c913e-65e8-441b-b40d-17216ac793d8" path="/var/lib/kubelet/pods/2b7c913e-65e8-441b-b40d-17216ac793d8/volumes"
Sep 06 19:30:53.853477 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:53.853445    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt"]
Sep 06 19:30:53.854057 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:53.853867    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b631e3c5-9c83-47ea-88d7-b02208086dcf" containerName="payload-pre-processing"
Sep 06 19:30:53.854057 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:53.853879    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b631e3c5-9c83-47ea-88d7-b02208086dcf" containerName="payload-pre-processing"
Sep 06 19:30:53.854057 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:53.853893    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e814b9c-be48-4eef-b6c2-370e7b20324c" containerName="main"
Sep 06 19:30:53.854057 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:53.853898    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e814b9c-be48-4eef-b6c2-370e7b20324c" containerName="main"
Sep 06 19:30:53.854057 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:53.853916    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29ce5aa5-2753-429d-bfb0-1493556b5999" containerName="payload-processing"
Sep 06 19:30:53.854057 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:53.853922    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="29ce5aa5-2753-429d-bfb0-1493556b5999" containerName="payload-processing"
Sep 06 19:30:53.854057 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:53.853928    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b7c913e-65e8-441b-b40d-17216ac793d8" containerName="main"
Sep 06 19:30:53.854057 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:53.853933    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b7c913e-65e8-441b-b40d-17216ac793d8" containerName="main"
Sep 06 19:30:53.854057 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:53.853990    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e814b9c-be48-4eef-b6c2-370e7b20324c" containerName="main"
Sep 06 19:30:53.854057 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:53.853999    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b7c913e-65e8-441b-b40d-17216ac793d8" containerName="main"
Sep 06 19:30:53.854057 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:53.854006    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="29ce5aa5-2753-429d-bfb0-1493556b5999" containerName="payload-processing"
Sep 06 19:30:53.854057 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:53.854013    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b631e3c5-9c83-47ea-88d7-b02208086dcf" containerName="payload-pre-processing"
Sep 06 19:30:53.858533 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:53.858515    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt"
Sep 06 19:30:53.861149 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:53.861131    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-c6813a-dockercfg-m79dk\""
Sep 06 19:30:53.861299 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:53.861281    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-c6813a\""
Sep 06 19:30:53.867152 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:53.867131    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt"]
Sep 06 19:30:53.966350 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:53.966314    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt\" (UID: \"36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt"
Sep 06 19:30:53.966491 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:53.966392    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ltwd\" (UniqueName: \"kubernetes.io/projected/36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9-kube-api-access-6ltwd\") pod \"payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt\" (UID: \"36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt"
Sep 06 19:30:54.067806 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:54.067773    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ltwd\" (UniqueName: \"kubernetes.io/projected/36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9-kube-api-access-6ltwd\") pod \"payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt\" (UID: \"36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt"
Sep 06 19:30:54.067950 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:54.067837    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt\" (UID: \"36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt"
Sep 06 19:30:54.068487 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:54.068466    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt\" (UID: \"36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt"
Sep 06 19:30:54.075436 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:54.075405    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ltwd\" (UniqueName: \"kubernetes.io/projected/36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9-kube-api-access-6ltwd\") pod \"payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt\" (UID: \"36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt"
Sep 06 19:30:54.169419 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:54.169353    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt"
Sep 06 19:30:54.439283 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:54.439214    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt"]
Sep 06 19:30:54.442817 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:30:54.442790    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36dc2d7e_8c9e_49e1_a0b6_0f9e264418a9.slice/crio-47c274df2b3e9bb925c0c548ebe554b0a10b7a07431e99c239fe88e8674febcb WatchSource:0}: Error finding container 47c274df2b3e9bb925c0c548ebe554b0a10b7a07431e99c239fe88e8674febcb: Status 404 returned error can't find the container with id 47c274df2b3e9bb925c0c548ebe554b0a10b7a07431e99c239fe88e8674febcb
Sep 06 19:30:54.836339 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:54.836306    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt" event={"ID":"36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9","Type":"ContainerStarted","Data":"db480a4e2cb2811efbe024fde2deb884daf4e337b2c54097ca423224151b7e61"}
Sep 06 19:30:54.836504 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:54.836358    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt" event={"ID":"36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9","Type":"ContainerStarted","Data":"47c274df2b3e9bb925c0c548ebe554b0a10b7a07431e99c239fe88e8674febcb"}
Sep 06 19:30:54.836504 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:54.836379    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt"
Sep 06 19:30:54.852275 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:54.852229    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt" podStartSLOduration=1.8522153270000001 podStartE2EDuration="1.852215327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:30:54.850137161 +0000 UTC m=+1189.031318971" watchObservedRunningTime="2026-09-06 19:30:54.852215327 +0000 UTC m=+1189.033397161"
Sep 06 19:30:58.452930 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:30:58.452899    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:31:05.845696 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:05.845664    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt"
Sep 06 19:31:06.383264 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:06.383232    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnssq_9254e6f1-b73d-49cc-8bc1-513925ccde4f/ovn-acl-logging/0.log"
Sep 06 19:31:06.386788 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:06.386764    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnssq_9254e6f1-b73d-49cc-8bc1-513925ccde4f/ovn-acl-logging/0.log"
Sep 06 19:31:10.053988 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:10.053954    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:31:19.843999 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:19.843968    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:31:29.343884 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.343851    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"]
Sep 06 19:31:29.348827 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.348808    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:29.352050 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.352027    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d73678\"/\"kube-root-ca.crt\""
Sep 06 19:31:29.352333 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.352174    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d73678\"/\"default-dockercfg-kjxhx\""
Sep 06 19:31:29.352333 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.352196    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d73678\"/\"test-model-d73678-kserve-self-signed-certs\""
Sep 06 19:31:29.352333 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.352268    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d73678\"/\"openshift-service-ca.crt\""
Sep 06 19:31:29.353857 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.353831    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"]
Sep 06 19:31:29.469809 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.469775    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-home\") pod \"test-model-d73678-kserve-6bb9cdbc75-95r6q\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") " pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:29.469976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.469827    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-tls-certs\") pod \"test-model-d73678-kserve-6bb9cdbc75-95r6q\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") " pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:29.469976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.469925    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-dshm\") pod \"test-model-d73678-kserve-6bb9cdbc75-95r6q\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") " pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:29.469976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.469955    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-model-cache\") pod \"test-model-d73678-kserve-6bb9cdbc75-95r6q\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") " pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:29.469976 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.469975    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l7sx\" (UniqueName: \"kubernetes.io/projected/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-kube-api-access-6l7sx\") pod \"test-model-d73678-kserve-6bb9cdbc75-95r6q\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") " pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:29.470163 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.470016    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-tmp-dir\") pod \"test-model-d73678-kserve-6bb9cdbc75-95r6q\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") " pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:29.571307 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.571283    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-dshm\") pod \"test-model-d73678-kserve-6bb9cdbc75-95r6q\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") " pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:29.571410 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.571315    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-model-cache\") pod \"test-model-d73678-kserve-6bb9cdbc75-95r6q\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") " pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:29.571410 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.571332    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l7sx\" (UniqueName: \"kubernetes.io/projected/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-kube-api-access-6l7sx\") pod \"test-model-d73678-kserve-6bb9cdbc75-95r6q\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") " pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:29.571410 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.571351    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-tmp-dir\") pod \"test-model-d73678-kserve-6bb9cdbc75-95r6q\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") " pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:29.571410 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.571371    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-home\") pod \"test-model-d73678-kserve-6bb9cdbc75-95r6q\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") " pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:29.571593 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.571567    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-tls-certs\") pod \"test-model-d73678-kserve-6bb9cdbc75-95r6q\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") " pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:29.571775 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.571758    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-home\") pod \"test-model-d73678-kserve-6bb9cdbc75-95r6q\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") " pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:29.571841 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.571790    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-model-cache\") pod \"test-model-d73678-kserve-6bb9cdbc75-95r6q\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") " pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:29.571893 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.571845    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-tmp-dir\") pod \"test-model-d73678-kserve-6bb9cdbc75-95r6q\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") " pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:29.573642 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.573598    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-dshm\") pod \"test-model-d73678-kserve-6bb9cdbc75-95r6q\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") " pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:29.574010 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.573994    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-tls-certs\") pod \"test-model-d73678-kserve-6bb9cdbc75-95r6q\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") " pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:29.578785 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.578763    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l7sx\" (UniqueName: \"kubernetes.io/projected/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-kube-api-access-6l7sx\") pod \"test-model-d73678-kserve-6bb9cdbc75-95r6q\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") " pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:29.659108 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.659021    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:29.928811 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:29.928788    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"]
Sep 06 19:31:29.930586 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:31:29.930557    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe80f4f1_77e1_4ffe_8c37_e43b87dbee12.slice/crio-5e0afe5d0e27a68fccc153ecdab8d0635f51ce4f1bddc08f5fbc8bd9b46a4253 WatchSource:0}: Error finding container 5e0afe5d0e27a68fccc153ecdab8d0635f51ce4f1bddc08f5fbc8bd9b46a4253: Status 404 returned error can't find the container with id 5e0afe5d0e27a68fccc153ecdab8d0635f51ce4f1bddc08f5fbc8bd9b46a4253
Sep 06 19:31:30.049186 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:30.049147    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q" event={"ID":"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12","Type":"ContainerStarted","Data":"99e0de02b705c51556b8ddbb113b1f142e7338973579838f475a48c4cf7ea9ee"}
Sep 06 19:31:30.049370 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:30.049198    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q" event={"ID":"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12","Type":"ContainerStarted","Data":"5e0afe5d0e27a68fccc153ecdab8d0635f51ce4f1bddc08f5fbc8bd9b46a4253"}
Sep 06 19:31:30.067207 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:30.067158    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q" podStartSLOduration=1.067142614 podStartE2EDuration="1.067142614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:31:30.064502735 +0000 UTC m=+1224.245684542" watchObservedRunningTime="2026-09-06 19:31:30.067142614 +0000 UTC m=+1224.248324672"
Sep 06 19:31:34.550676 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:34.550641    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:31:39.659795 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:39.659747    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:39.660224 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:39.659808    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:39.671935 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:39.671913    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:40.117091 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:40.117062    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:31:45.077459 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.077425    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2"]
Sep 06 19:31:45.083260 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.083231    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2"
Sep 06 19:31:45.086659 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.086633    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-032bc360-dockercfg-j5mww\""
Sep 06 19:31:45.086809 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.086688    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-032bc360\""
Sep 06 19:31:45.091965 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.091942    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2"]
Sep 06 19:31:45.110052 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.110027    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9"]
Sep 06 19:31:45.113722 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.113706    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9"
Sep 06 19:31:45.123931 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.123910    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9"]
Sep 06 19:31:45.210162 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.210131    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn2mk\" (UniqueName: \"kubernetes.io/projected/884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e-kube-api-access-sn2mk\") pod \"payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2\" (UID: \"884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2"
Sep 06 19:31:45.210292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.210168    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8-plugins-config-volume\") pod \"payload-processing-e2e-mt-032bc360-67df587857-6w6c9\" (UID: \"ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8\") " pod="openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9"
Sep 06 19:31:45.210292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.210194    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2\" (UID: \"884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2"
Sep 06 19:31:45.210292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.210263    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5gbs\" (UniqueName: \"kubernetes.io/projected/ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8-kube-api-access-h5gbs\") pod \"payload-processing-e2e-mt-032bc360-67df587857-6w6c9\" (UID: \"ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8\") " pod="openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9"
Sep 06 19:31:45.310809 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.310778    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn2mk\" (UniqueName: \"kubernetes.io/projected/884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e-kube-api-access-sn2mk\") pod \"payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2\" (UID: \"884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2"
Sep 06 19:31:45.310928 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.310819    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8-plugins-config-volume\") pod \"payload-processing-e2e-mt-032bc360-67df587857-6w6c9\" (UID: \"ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8\") " pod="openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9"
Sep 06 19:31:45.310928 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.310851    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2\" (UID: \"884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2"
Sep 06 19:31:45.310928 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.310881    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5gbs\" (UniqueName: \"kubernetes.io/projected/ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8-kube-api-access-h5gbs\") pod \"payload-processing-e2e-mt-032bc360-67df587857-6w6c9\" (UID: \"ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8\") " pod="openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9"
Sep 06 19:31:45.311475 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.311456    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8-plugins-config-volume\") pod \"payload-processing-e2e-mt-032bc360-67df587857-6w6c9\" (UID: \"ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8\") " pod="openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9"
Sep 06 19:31:45.311534 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.311480    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2\" (UID: \"884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2"
Sep 06 19:31:45.322542 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.322522    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5gbs\" (UniqueName: \"kubernetes.io/projected/ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8-kube-api-access-h5gbs\") pod \"payload-processing-e2e-mt-032bc360-67df587857-6w6c9\" (UID: \"ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8\") " pod="openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9"
Sep 06 19:31:45.322663 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.322624    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn2mk\" (UniqueName: \"kubernetes.io/projected/884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e-kube-api-access-sn2mk\") pod \"payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2\" (UID: \"884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2"
Sep 06 19:31:45.397552 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.397496    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2"
Sep 06 19:31:45.423862 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.423839    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9"
Sep 06 19:31:45.670759 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.670735    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2"]
Sep 06 19:31:45.672044 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:31:45.672020    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884d9b4f_c3ee_403d_bfb6_9157ad3e6d1e.slice/crio-4f59c4530b8859177be579ef29eabcdfb973b098a884ee2e4885067415650a3d WatchSource:0}: Error finding container 4f59c4530b8859177be579ef29eabcdfb973b098a884ee2e4885067415650a3d: Status 404 returned error can't find the container with id 4f59c4530b8859177be579ef29eabcdfb973b098a884ee2e4885067415650a3d
Sep 06 19:31:45.701549 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:45.701525    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9"]
Sep 06 19:31:45.709725 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:31:45.709700    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6e9c70_1514_4c17_a5d1_8ab1e8fd8ce8.slice/crio-5a57a8c2324fcb2d4a97cfb852658c9e0b6036fd5ad2a81a3db70b8d5d6acce0 WatchSource:0}: Error finding container 5a57a8c2324fcb2d4a97cfb852658c9e0b6036fd5ad2a81a3db70b8d5d6acce0: Status 404 returned error can't find the container with id 5a57a8c2324fcb2d4a97cfb852658c9e0b6036fd5ad2a81a3db70b8d5d6acce0
Sep 06 19:31:46.142988 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:46.142948    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2" event={"ID":"884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e","Type":"ContainerStarted","Data":"af04d6d27c1d7816049219dd4cb120bf4c63d181802501edf781c72010de75f0"}
Sep 06 19:31:46.143418 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:46.143020    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2"
Sep 06 19:31:46.143418 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:46.143049    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2" event={"ID":"884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e","Type":"ContainerStarted","Data":"4f59c4530b8859177be579ef29eabcdfb973b098a884ee2e4885067415650a3d"}
Sep 06 19:31:46.144809 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:46.144778    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9" event={"ID":"ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8","Type":"ContainerStarted","Data":"2e1d7d94aeda7b2a72cbf33baad6cdd2c356a0ee3d2a89b64beb2e84365f21c8"}
Sep 06 19:31:46.144935 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:46.144834    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9" event={"ID":"ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8","Type":"ContainerStarted","Data":"5a57a8c2324fcb2d4a97cfb852658c9e0b6036fd5ad2a81a3db70b8d5d6acce0"}
Sep 06 19:31:46.144935 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:46.144861    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9"
Sep 06 19:31:46.157934 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:46.157896    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2" podStartSLOduration=1.157883556 podStartE2EDuration="1.157883556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:31:46.156740017 +0000 UTC m=+1240.337921822" watchObservedRunningTime="2026-09-06 19:31:46.157883556 +0000 UTC m=+1240.339065362"
Sep 06 19:31:46.171514 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:46.171471    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9" podStartSLOduration=1.1714572269999999 podStartE2EDuration="1.171457227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:31:46.170371944 +0000 UTC m=+1240.351553750" watchObservedRunningTime="2026-09-06 19:31:46.171457227 +0000 UTC m=+1240.352639032"
Sep 06 19:31:49.127218 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.127187    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"]
Sep 06 19:31:49.130480 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.130465    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:31:49.134169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.134148    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c6813a\"/\"openshift-service-ca.crt\""
Sep 06 19:31:49.134413 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.134157    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c6813a\"/\"default-dockercfg-h45c4\""
Sep 06 19:31:49.134528 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.134155    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c6813a\"/\"kube-root-ca.crt\""
Sep 06 19:31:49.134528 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.134185    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c6813a\"/\"explicit-model-c6813a-kserve-self-signed-certs\""
Sep 06 19:31:49.137736 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.137716    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"]
Sep 06 19:31:49.243436 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.243412    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b06c5654-f070-42f7-9edf-2a9f75154926-tls-certs\") pod \"explicit-model-c6813a-kserve-87cfdd977-6nqft\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") " pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:31:49.243598 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.243455    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-tmp-dir\") pod \"explicit-model-c6813a-kserve-87cfdd977-6nqft\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") " pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:31:49.243598 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.243480    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml9xc\" (UniqueName: \"kubernetes.io/projected/b06c5654-f070-42f7-9edf-2a9f75154926-kube-api-access-ml9xc\") pod \"explicit-model-c6813a-kserve-87cfdd977-6nqft\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") " pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:31:49.243598 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.243536    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-dshm\") pod \"explicit-model-c6813a-kserve-87cfdd977-6nqft\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") " pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:31:49.243598 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.243595    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-model-cache\") pod \"explicit-model-c6813a-kserve-87cfdd977-6nqft\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") " pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:31:49.243820 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.243652    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-home\") pod \"explicit-model-c6813a-kserve-87cfdd977-6nqft\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") " pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:31:49.344776 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.344740    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml9xc\" (UniqueName: \"kubernetes.io/projected/b06c5654-f070-42f7-9edf-2a9f75154926-kube-api-access-ml9xc\") pod \"explicit-model-c6813a-kserve-87cfdd977-6nqft\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") " pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:31:49.344929 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.344784    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-dshm\") pod \"explicit-model-c6813a-kserve-87cfdd977-6nqft\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") " pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:31:49.344998 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.344928    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-model-cache\") pod \"explicit-model-c6813a-kserve-87cfdd977-6nqft\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") " pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:31:49.344998 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.344979    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-home\") pod \"explicit-model-c6813a-kserve-87cfdd977-6nqft\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") " pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:31:49.345104 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.345065    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b06c5654-f070-42f7-9edf-2a9f75154926-tls-certs\") pod \"explicit-model-c6813a-kserve-87cfdd977-6nqft\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") " pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:31:49.345159 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.345122    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-tmp-dir\") pod \"explicit-model-c6813a-kserve-87cfdd977-6nqft\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") " pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:31:49.345307 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.345287    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-model-cache\") pod \"explicit-model-c6813a-kserve-87cfdd977-6nqft\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") " pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:31:49.345397 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.345375    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-home\") pod \"explicit-model-c6813a-kserve-87cfdd977-6nqft\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") " pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:31:49.345457 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.345417    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-tmp-dir\") pod \"explicit-model-c6813a-kserve-87cfdd977-6nqft\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") " pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:31:49.347082 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.347063    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-dshm\") pod \"explicit-model-c6813a-kserve-87cfdd977-6nqft\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") " pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:31:49.347528 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.347508    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b06c5654-f070-42f7-9edf-2a9f75154926-tls-certs\") pod \"explicit-model-c6813a-kserve-87cfdd977-6nqft\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") " pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:31:49.351811 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.351789    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml9xc\" (UniqueName: \"kubernetes.io/projected/b06c5654-f070-42f7-9edf-2a9f75154926-kube-api-access-ml9xc\") pod \"explicit-model-c6813a-kserve-87cfdd977-6nqft\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") " pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:31:49.441742 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.441649    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:31:49.712940 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:49.712916    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"]
Sep 06 19:31:49.714895 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:31:49.714870    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb06c5654_f070_42f7_9edf_2a9f75154926.slice/crio-0f54afc4ebf446d282153e76b3ee1e3300ea2808aed47b0c6ed74a65a3d5a625 WatchSource:0}: Error finding container 0f54afc4ebf446d282153e76b3ee1e3300ea2808aed47b0c6ed74a65a3d5a625: Status 404 returned error can't find the container with id 0f54afc4ebf446d282153e76b3ee1e3300ea2808aed47b0c6ed74a65a3d5a625
Sep 06 19:31:50.170308 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:50.170219    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft" event={"ID":"b06c5654-f070-42f7-9edf-2a9f75154926","Type":"ContainerStarted","Data":"ba86bacfed8b06a2f62b16552f3bd085dc2d207b19a43f29885512f48b1b756d"}
Sep 06 19:31:50.170308 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:50.170256    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft" event={"ID":"b06c5654-f070-42f7-9edf-2a9f75154926","Type":"ContainerStarted","Data":"0f54afc4ebf446d282153e76b3ee1e3300ea2808aed47b0c6ed74a65a3d5a625"}
Sep 06 19:31:50.184684 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:50.184636    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft" podStartSLOduration=1.1845906720000001 podStartE2EDuration="1.184590672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:31:50.184592178 +0000 UTC m=+1244.365773986" watchObservedRunningTime="2026-09-06 19:31:50.184590672 +0000 UTC m=+1244.365772469"
Sep 06 19:31:53.345920 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:53.345875    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:31:57.153291 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:57.153261    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9"
Sep 06 19:31:57.153750 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:57.153662    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2"
Sep 06 19:31:59.442416 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:59.442384    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:31:59.442896 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:59.442461    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:31:59.455521 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:31:59.455491    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:32:00.241380 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:00.241347    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:32:00.931062 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:00.931030    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"]
Sep 06 19:32:01.357262 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:32:01.357228    2566 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-self-signed-certs: secret "explicit-model-c6813a-kserve-self-signed-certs" not found
Sep 06 19:32:01.357440 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:32:01.357314    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b06c5654-f070-42f7-9edf-2a9f75154926-tls-certs podName:b06c5654-f070-42f7-9edf-2a9f75154926 nodeName:}" failed. No retries permitted until 2026-09-06 19:32:01.857291553 +0000 UTC m=+1256.038473338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b06c5654-f070-42f7-9edf-2a9f75154926-tls-certs") pod "explicit-model-c6813a-kserve-87cfdd977-6nqft" (UID: "b06c5654-f070-42f7-9edf-2a9f75154926") : secret "explicit-model-c6813a-kserve-self-signed-certs" not found
Sep 06 19:32:01.861519 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:32:01.861486    2566 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-self-signed-certs: secret "explicit-model-c6813a-kserve-self-signed-certs" not found
Sep 06 19:32:01.861713 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:32:01.861564    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b06c5654-f070-42f7-9edf-2a9f75154926-tls-certs podName:b06c5654-f070-42f7-9edf-2a9f75154926 nodeName:}" failed. No retries permitted until 2026-09-06 19:32:02.861546348 +0000 UTC m=+1257.042728132 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b06c5654-f070-42f7-9edf-2a9f75154926-tls-certs") pod "explicit-model-c6813a-kserve-87cfdd977-6nqft" (UID: "b06c5654-f070-42f7-9edf-2a9f75154926") : secret "explicit-model-c6813a-kserve-self-signed-certs" not found
Sep 06 19:32:02.871143 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:32:02.871113    2566 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-self-signed-certs: secret "explicit-model-c6813a-kserve-self-signed-certs" not found
Sep 06 19:32:02.871557 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:32:02.871176    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b06c5654-f070-42f7-9edf-2a9f75154926-tls-certs podName:b06c5654-f070-42f7-9edf-2a9f75154926 nodeName:}" failed. No retries permitted until 2026-09-06 19:32:04.871161772 +0000 UTC m=+1259.052343556 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b06c5654-f070-42f7-9edf-2a9f75154926-tls-certs") pod "explicit-model-c6813a-kserve-87cfdd977-6nqft" (UID: "b06c5654-f070-42f7-9edf-2a9f75154926") : secret "explicit-model-c6813a-kserve-self-signed-certs" not found
Sep 06 19:32:04.891134 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:32:04.891106    2566 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-self-signed-certs: secret "explicit-model-c6813a-kserve-self-signed-certs" not found
Sep 06 19:32:04.891485 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:32:04.891180    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b06c5654-f070-42f7-9edf-2a9f75154926-tls-certs podName:b06c5654-f070-42f7-9edf-2a9f75154926 nodeName:}" failed. No retries permitted until 2026-09-06 19:32:08.891164863 +0000 UTC m=+1263.072346650 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b06c5654-f070-42f7-9edf-2a9f75154926-tls-certs") pod "explicit-model-c6813a-kserve-87cfdd977-6nqft" (UID: "b06c5654-f070-42f7-9edf-2a9f75154926") : secret "explicit-model-c6813a-kserve-self-signed-certs" not found
Sep 06 19:32:08.921080 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:32:08.921049    2566 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-self-signed-certs: secret "explicit-model-c6813a-kserve-self-signed-certs" not found
Sep 06 19:32:08.921435 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:32:08.921117    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b06c5654-f070-42f7-9edf-2a9f75154926-tls-certs podName:b06c5654-f070-42f7-9edf-2a9f75154926 nodeName:}" failed. No retries permitted until 2026-09-06 19:32:16.9211011 +0000 UTC m=+1271.102282884 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b06c5654-f070-42f7-9edf-2a9f75154926-tls-certs") pod "explicit-model-c6813a-kserve-87cfdd977-6nqft" (UID: "b06c5654-f070-42f7-9edf-2a9f75154926") : secret "explicit-model-c6813a-kserve-self-signed-certs" not found
Sep 06 19:32:11.720836 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:11.720803    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt"]
Sep 06 19:32:11.721303 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:11.721046    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt" podUID="36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9" containerName="payload-pre-processing" containerID="cri-o://db480a4e2cb2811efbe024fde2deb884daf4e337b2c54097ca423224151b7e61" gracePeriod=30
Sep 06 19:32:11.977168 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:11.977114    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt"
Sep 06 19:32:12.046128 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:12.046099    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9-plugins-config-volume\") pod \"36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9\" (UID: \"36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9\") "
Sep 06 19:32:12.046128 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:12.046134    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ltwd\" (UniqueName: \"kubernetes.io/projected/36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9-kube-api-access-6ltwd\") pod \"36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9\" (UID: \"36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9\") "
Sep 06 19:32:12.046424 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:12.046404    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9" (UID: "36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:32:12.048404 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:12.048383    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9-kube-api-access-6ltwd" (OuterVolumeSpecName: "kube-api-access-6ltwd") pod "36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9" (UID: "36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9"). InnerVolumeSpecName "kube-api-access-6ltwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:32:12.147268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:12.147236    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9-plugins-config-volume\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:32:12.147268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:12.147262    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ltwd\" (UniqueName: \"kubernetes.io/projected/36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9-kube-api-access-6ltwd\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:32:12.302051 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:12.301968    2566 generic.go:356] "Generic (PLEG): container finished" podID="36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9" containerID="db480a4e2cb2811efbe024fde2deb884daf4e337b2c54097ca423224151b7e61" exitCode=0
Sep 06 19:32:12.302051 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:12.302011    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt" event={"ID":"36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9","Type":"ContainerDied","Data":"db480a4e2cb2811efbe024fde2deb884daf4e337b2c54097ca423224151b7e61"}
Sep 06 19:32:12.302051 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:12.302035    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt"
Sep 06 19:32:12.302283 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:12.302061    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt" event={"ID":"36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9","Type":"ContainerDied","Data":"47c274df2b3e9bb925c0c548ebe554b0a10b7a07431e99c239fe88e8674febcb"}
Sep 06 19:32:12.302283 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:12.302088    2566 scope.go:117] "RemoveContainer" containerID="db480a4e2cb2811efbe024fde2deb884daf4e337b2c54097ca423224151b7e61"
Sep 06 19:32:12.313977 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:12.313940    2566 scope.go:117] "RemoveContainer" containerID="db480a4e2cb2811efbe024fde2deb884daf4e337b2c54097ca423224151b7e61"
Sep 06 19:32:12.314311 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:32:12.314265    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db480a4e2cb2811efbe024fde2deb884daf4e337b2c54097ca423224151b7e61\": container with ID starting with db480a4e2cb2811efbe024fde2deb884daf4e337b2c54097ca423224151b7e61 not found: ID does not exist" containerID="db480a4e2cb2811efbe024fde2deb884daf4e337b2c54097ca423224151b7e61"
Sep 06 19:32:12.314406 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:12.314306    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db480a4e2cb2811efbe024fde2deb884daf4e337b2c54097ca423224151b7e61"} err="failed to get container status \"db480a4e2cb2811efbe024fde2deb884daf4e337b2c54097ca423224151b7e61\": rpc error: code = NotFound desc = could not find container \"db480a4e2cb2811efbe024fde2deb884daf4e337b2c54097ca423224151b7e61\": container with ID starting with db480a4e2cb2811efbe024fde2deb884daf4e337b2c54097ca423224151b7e61 not found: ID does not exist"
Sep 06 19:32:12.329008 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:12.328982    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt"]
Sep 06 19:32:12.331592 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:12.331573    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c6813a-cf649b7c8-g2ggt"]
Sep 06 19:32:14.316919 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:14.316884    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9" path="/var/lib/kubelet/pods/36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9/volumes"
Sep 06 19:32:14.850025 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:14.849992    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:32:16.988178 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:32:16.988149    2566 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-self-signed-certs: secret "explicit-model-c6813a-kserve-self-signed-certs" not found
Sep 06 19:32:16.988672 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:32:16.988229    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b06c5654-f070-42f7-9edf-2a9f75154926-tls-certs podName:b06c5654-f070-42f7-9edf-2a9f75154926 nodeName:}" failed. No retries permitted until 2026-09-06 19:32:32.988210089 +0000 UTC m=+1287.169391873 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b06c5654-f070-42f7-9edf-2a9f75154926-tls-certs") pod "explicit-model-c6813a-kserve-87cfdd977-6nqft" (UID: "b06c5654-f070-42f7-9edf-2a9f75154926") : secret "explicit-model-c6813a-kserve-self-signed-certs" not found
Sep 06 19:32:17.263572 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:17.263467    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft" podUID="b06c5654-f070-42f7-9edf-2a9f75154926" containerName="main" containerID="cri-o://ba86bacfed8b06a2f62b16552f3bd085dc2d207b19a43f29885512f48b1b756d" gracePeriod=45
Sep 06 19:32:17.512324 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:17.512297    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:32:17.592987 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:17.592922    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b06c5654-f070-42f7-9edf-2a9f75154926-tls-certs\") pod \"b06c5654-f070-42f7-9edf-2a9f75154926\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") "
Sep 06 19:32:17.592987 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:17.592956    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-tmp-dir\") pod \"b06c5654-f070-42f7-9edf-2a9f75154926\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") "
Sep 06 19:32:17.593186 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:17.593005    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-model-cache\") pod \"b06c5654-f070-42f7-9edf-2a9f75154926\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") "
Sep 06 19:32:17.593186 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:17.593026    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml9xc\" (UniqueName: \"kubernetes.io/projected/b06c5654-f070-42f7-9edf-2a9f75154926-kube-api-access-ml9xc\") pod \"b06c5654-f070-42f7-9edf-2a9f75154926\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") "
Sep 06 19:32:17.593186 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:17.593045    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-home\") pod \"b06c5654-f070-42f7-9edf-2a9f75154926\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") "
Sep 06 19:32:17.593186 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:17.593060    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-dshm\") pod \"b06c5654-f070-42f7-9edf-2a9f75154926\" (UID: \"b06c5654-f070-42f7-9edf-2a9f75154926\") "
Sep 06 19:32:17.593384 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:17.593297    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-model-cache" (OuterVolumeSpecName: "model-cache") pod "b06c5654-f070-42f7-9edf-2a9f75154926" (UID: "b06c5654-f070-42f7-9edf-2a9f75154926"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:32:17.593384 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:17.593309    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-home" (OuterVolumeSpecName: "home") pod "b06c5654-f070-42f7-9edf-2a9f75154926" (UID: "b06c5654-f070-42f7-9edf-2a9f75154926"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:32:17.593384 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:17.593324    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b06c5654-f070-42f7-9edf-2a9f75154926" (UID: "b06c5654-f070-42f7-9edf-2a9f75154926"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:32:17.593550 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:17.593429    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-tmp-dir\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:32:17.593550 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:17.593448    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-model-cache\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:32:17.593550 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:17.593460    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-home\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:32:17.595385 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:17.595362    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b06c5654-f070-42f7-9edf-2a9f75154926-kube-api-access-ml9xc" (OuterVolumeSpecName: "kube-api-access-ml9xc") pod "b06c5654-f070-42f7-9edf-2a9f75154926" (UID: "b06c5654-f070-42f7-9edf-2a9f75154926"). InnerVolumeSpecName "kube-api-access-ml9xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:32:17.595385 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:17.595362    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-dshm" (OuterVolumeSpecName: "dshm") pod "b06c5654-f070-42f7-9edf-2a9f75154926" (UID: "b06c5654-f070-42f7-9edf-2a9f75154926"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:32:17.595490 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:17.595417    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b06c5654-f070-42f7-9edf-2a9f75154926-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b06c5654-f070-42f7-9edf-2a9f75154926" (UID: "b06c5654-f070-42f7-9edf-2a9f75154926"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:32:17.694832 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:17.694798    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b06c5654-f070-42f7-9edf-2a9f75154926-tls-certs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:32:17.694832 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:17.694829    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml9xc\" (UniqueName: \"kubernetes.io/projected/b06c5654-f070-42f7-9edf-2a9f75154926-kube-api-access-ml9xc\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:32:17.694832 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:17.694839    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b06c5654-f070-42f7-9edf-2a9f75154926-dshm\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:32:18.342028 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:18.341993    2566 generic.go:356] "Generic (PLEG): container finished" podID="b06c5654-f070-42f7-9edf-2a9f75154926" containerID="ba86bacfed8b06a2f62b16552f3bd085dc2d207b19a43f29885512f48b1b756d" exitCode=0
Sep 06 19:32:18.342344 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:18.342032    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"
Sep 06 19:32:18.342344 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:18.342084    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft" event={"ID":"b06c5654-f070-42f7-9edf-2a9f75154926","Type":"ContainerDied","Data":"ba86bacfed8b06a2f62b16552f3bd085dc2d207b19a43f29885512f48b1b756d"}
Sep 06 19:32:18.342344 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:18.342120    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft" event={"ID":"b06c5654-f070-42f7-9edf-2a9f75154926","Type":"ContainerDied","Data":"0f54afc4ebf446d282153e76b3ee1e3300ea2808aed47b0c6ed74a65a3d5a625"}
Sep 06 19:32:18.342344 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:18.342140    2566 scope.go:117] "RemoveContainer" containerID="ba86bacfed8b06a2f62b16552f3bd085dc2d207b19a43f29885512f48b1b756d"
Sep 06 19:32:18.351445 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:18.351415    2566 scope.go:117] "RemoveContainer" containerID="ba86bacfed8b06a2f62b16552f3bd085dc2d207b19a43f29885512f48b1b756d"
Sep 06 19:32:18.351750 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:32:18.351733    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba86bacfed8b06a2f62b16552f3bd085dc2d207b19a43f29885512f48b1b756d\": container with ID starting with ba86bacfed8b06a2f62b16552f3bd085dc2d207b19a43f29885512f48b1b756d not found: ID does not exist" containerID="ba86bacfed8b06a2f62b16552f3bd085dc2d207b19a43f29885512f48b1b756d"
Sep 06 19:32:18.351815 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:18.351764    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba86bacfed8b06a2f62b16552f3bd085dc2d207b19a43f29885512f48b1b756d"} err="failed to get container status \"ba86bacfed8b06a2f62b16552f3bd085dc2d207b19a43f29885512f48b1b756d\": rpc error: code = NotFound desc = could not find container \"ba86bacfed8b06a2f62b16552f3bd085dc2d207b19a43f29885512f48b1b756d\": container with ID starting with ba86bacfed8b06a2f62b16552f3bd085dc2d207b19a43f29885512f48b1b756d not found: ID does not exist"
Sep 06 19:32:18.360704 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:18.360682    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"]
Sep 06 19:32:18.366361 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:18.366340    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-c6813a/explicit-model-c6813a-kserve-87cfdd977-6nqft"]
Sep 06 19:32:20.311134 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:20.311096    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b06c5654-f070-42f7-9edf-2a9f75154926" path="/var/lib/kubelet/pods/b06c5654-f070-42f7-9edf-2a9f75154926/volumes"
Sep 06 19:32:25.930646 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:25.930583    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9"]
Sep 06 19:32:25.931103 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:25.930900    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9" podUID="ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8" containerName="payload-processing" containerID="cri-o://2e1d7d94aeda7b2a72cbf33baad6cdd2c356a0ee3d2a89b64beb2e84365f21c8" gracePeriod=30
Sep 06 19:32:25.939186 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:25.939162    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2"]
Sep 06 19:32:25.939394 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:25.939371    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2" podUID="884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e" containerName="payload-pre-processing" containerID="cri-o://af04d6d27c1d7816049219dd4cb120bf4c63d181802501edf781c72010de75f0" gracePeriod=30
Sep 06 19:32:26.210882 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.210859    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9"
Sep 06 19:32:26.213805 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.213782    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2"
Sep 06 19:32:26.276843 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.276815    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8-plugins-config-volume\") pod \"ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8\" (UID: \"ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8\") "
Sep 06 19:32:26.277011 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.276881    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e-plugins-config-volume\") pod \"884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e\" (UID: \"884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e\") "
Sep 06 19:32:26.277011 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.276944    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sn2mk\" (UniqueName: \"kubernetes.io/projected/884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e-kube-api-access-sn2mk\") pod \"884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e\" (UID: \"884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e\") "
Sep 06 19:32:26.277083 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.277026    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5gbs\" (UniqueName: \"kubernetes.io/projected/ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8-kube-api-access-h5gbs\") pod \"ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8\" (UID: \"ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8\") "
Sep 06 19:32:26.277281 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.277256    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8" (UID: "ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:32:26.277334 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.277292    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e" (UID: "884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:32:26.279133 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.279110    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8-kube-api-access-h5gbs" (OuterVolumeSpecName: "kube-api-access-h5gbs") pod "ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8" (UID: "ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8"). InnerVolumeSpecName "kube-api-access-h5gbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:32:26.279212 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.279146    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e-kube-api-access-sn2mk" (OuterVolumeSpecName: "kube-api-access-sn2mk") pod "884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e" (UID: "884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e"). InnerVolumeSpecName "kube-api-access-sn2mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:32:26.378033 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.378009    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8-plugins-config-volume\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:32:26.378033 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.378030    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e-plugins-config-volume\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:32:26.378164 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.378039    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn2mk\" (UniqueName: \"kubernetes.io/projected/884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e-kube-api-access-sn2mk\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:32:26.378164 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.378047    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5gbs\" (UniqueName: \"kubernetes.io/projected/ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8-kube-api-access-h5gbs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:32:26.388652 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.388623    2566 generic.go:356] "Generic (PLEG): container finished" podID="884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e" containerID="af04d6d27c1d7816049219dd4cb120bf4c63d181802501edf781c72010de75f0" exitCode=0
Sep 06 19:32:26.388752 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.388682    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2"
Sep 06 19:32:26.388752 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.388723    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2" event={"ID":"884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e","Type":"ContainerDied","Data":"af04d6d27c1d7816049219dd4cb120bf4c63d181802501edf781c72010de75f0"}
Sep 06 19:32:26.388846 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.388768    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2" event={"ID":"884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e","Type":"ContainerDied","Data":"4f59c4530b8859177be579ef29eabcdfb973b098a884ee2e4885067415650a3d"}
Sep 06 19:32:26.388846 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.388791    2566 scope.go:117] "RemoveContainer" containerID="af04d6d27c1d7816049219dd4cb120bf4c63d181802501edf781c72010de75f0"
Sep 06 19:32:26.390231 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.390207    2566 generic.go:356] "Generic (PLEG): container finished" podID="ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8" containerID="2e1d7d94aeda7b2a72cbf33baad6cdd2c356a0ee3d2a89b64beb2e84365f21c8" exitCode=0
Sep 06 19:32:26.390341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.390272    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9"
Sep 06 19:32:26.390341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.390303    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9" event={"ID":"ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8","Type":"ContainerDied","Data":"2e1d7d94aeda7b2a72cbf33baad6cdd2c356a0ee3d2a89b64beb2e84365f21c8"}
Sep 06 19:32:26.390446 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.390343    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9" event={"ID":"ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8","Type":"ContainerDied","Data":"5a57a8c2324fcb2d4a97cfb852658c9e0b6036fd5ad2a81a3db70b8d5d6acce0"}
Sep 06 19:32:26.398434 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.398417    2566 scope.go:117] "RemoveContainer" containerID="af04d6d27c1d7816049219dd4cb120bf4c63d181802501edf781c72010de75f0"
Sep 06 19:32:26.398724 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:32:26.398706    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af04d6d27c1d7816049219dd4cb120bf4c63d181802501edf781c72010de75f0\": container with ID starting with af04d6d27c1d7816049219dd4cb120bf4c63d181802501edf781c72010de75f0 not found: ID does not exist" containerID="af04d6d27c1d7816049219dd4cb120bf4c63d181802501edf781c72010de75f0"
Sep 06 19:32:26.398774 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.398731    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af04d6d27c1d7816049219dd4cb120bf4c63d181802501edf781c72010de75f0"} err="failed to get container status \"af04d6d27c1d7816049219dd4cb120bf4c63d181802501edf781c72010de75f0\": rpc error: code = NotFound desc = could not find container \"af04d6d27c1d7816049219dd4cb120bf4c63d181802501edf781c72010de75f0\": container with ID starting with af04d6d27c1d7816049219dd4cb120bf4c63d181802501edf781c72010de75f0 not found: ID does not exist"
Sep 06 19:32:26.398774 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.398748    2566 scope.go:117] "RemoveContainer" containerID="2e1d7d94aeda7b2a72cbf33baad6cdd2c356a0ee3d2a89b64beb2e84365f21c8"
Sep 06 19:32:26.406786 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.406766    2566 scope.go:117] "RemoveContainer" containerID="2e1d7d94aeda7b2a72cbf33baad6cdd2c356a0ee3d2a89b64beb2e84365f21c8"
Sep 06 19:32:26.407062 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:32:26.407044    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e1d7d94aeda7b2a72cbf33baad6cdd2c356a0ee3d2a89b64beb2e84365f21c8\": container with ID starting with 2e1d7d94aeda7b2a72cbf33baad6cdd2c356a0ee3d2a89b64beb2e84365f21c8 not found: ID does not exist" containerID="2e1d7d94aeda7b2a72cbf33baad6cdd2c356a0ee3d2a89b64beb2e84365f21c8"
Sep 06 19:32:26.407130 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.407071    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e1d7d94aeda7b2a72cbf33baad6cdd2c356a0ee3d2a89b64beb2e84365f21c8"} err="failed to get container status \"2e1d7d94aeda7b2a72cbf33baad6cdd2c356a0ee3d2a89b64beb2e84365f21c8\": rpc error: code = NotFound desc = could not find container \"2e1d7d94aeda7b2a72cbf33baad6cdd2c356a0ee3d2a89b64beb2e84365f21c8\": container with ID starting with 2e1d7d94aeda7b2a72cbf33baad6cdd2c356a0ee3d2a89b64beb2e84365f21c8 not found: ID does not exist"
Sep 06 19:32:26.408000 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.407982    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2"]
Sep 06 19:32:26.412814 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.412795    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-032bc360-cf8c48f9d-bnpg2"]
Sep 06 19:32:26.422505 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.422482    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9"]
Sep 06 19:32:26.425075 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:26.425058    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-032bc360-67df587857-6w6c9"]
Sep 06 19:32:28.310054 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:28.310025    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e" path="/var/lib/kubelet/pods/884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e/volumes"
Sep 06 19:32:28.310478 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:28.310348    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8" path="/var/lib/kubelet/pods/ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8/volumes"
Sep 06 19:32:29.844540 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:29.844506    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:32:39.347888 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.347851    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"]
Sep 06 19:32:39.348314 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.348242    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9" containerName="payload-pre-processing"
Sep 06 19:32:39.348314 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.348254    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9" containerName="payload-pre-processing"
Sep 06 19:32:39.348314 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.348262    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e" containerName="payload-pre-processing"
Sep 06 19:32:39.348314 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.348267    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e" containerName="payload-pre-processing"
Sep 06 19:32:39.348314 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.348284    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8" containerName="payload-processing"
Sep 06 19:32:39.348314 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.348289    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8" containerName="payload-processing"
Sep 06 19:32:39.348314 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.348295    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b06c5654-f070-42f7-9edf-2a9f75154926" containerName="main"
Sep 06 19:32:39.348314 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.348300    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b06c5654-f070-42f7-9edf-2a9f75154926" containerName="main"
Sep 06 19:32:39.348557 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.348364    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b06c5654-f070-42f7-9edf-2a9f75154926" containerName="main"
Sep 06 19:32:39.348557 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.348375    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="36dc2d7e-8c9e-49e1-a0b6-0f9e264418a9" containerName="payload-pre-processing"
Sep 06 19:32:39.348557 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.348381    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec6e9c70-1514-4c17-a5d1-8ab1e8fd8ce8" containerName="payload-processing"
Sep 06 19:32:39.348557 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.348387    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="884d9b4f-c3ee-403d-bfb6-9157ad3e6d1e" containerName="payload-pre-processing"
Sep 06 19:32:39.351635 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.351616    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"
Sep 06 19:32:39.355071 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.355046    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-477eea\"/\"noten-model-477eea-kserve-self-signed-certs\""
Sep 06 19:32:39.355179 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.355048    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-477eea\"/\"openshift-service-ca.crt\""
Sep 06 19:32:39.355179 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.355050    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-477eea\"/\"default-dockercfg-tcdtc\""
Sep 06 19:32:39.355179 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.355051    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-477eea\"/\"kube-root-ca.crt\""
Sep 06 19:32:39.360943 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.360922    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"]
Sep 06 19:32:39.489307 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.489279    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6cb4749-1b55-4a9e-a867-0b51dce53855-tls-certs\") pod \"noten-model-477eea-kserve-7457f98d7f-vbbxx\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") " pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"
Sep 06 19:32:39.489465 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.489321    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-tmp-dir\") pod \"noten-model-477eea-kserve-7457f98d7f-vbbxx\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") " pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"
Sep 06 19:32:39.489465 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.489349    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-dshm\") pod \"noten-model-477eea-kserve-7457f98d7f-vbbxx\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") " pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"
Sep 06 19:32:39.489465 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.489413    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-model-cache\") pod \"noten-model-477eea-kserve-7457f98d7f-vbbxx\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") " pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"
Sep 06 19:32:39.489465 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.489449    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tb8x\" (UniqueName: \"kubernetes.io/projected/e6cb4749-1b55-4a9e-a867-0b51dce53855-kube-api-access-9tb8x\") pod \"noten-model-477eea-kserve-7457f98d7f-vbbxx\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") " pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"
Sep 06 19:32:39.489628 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.489559    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-home\") pod \"noten-model-477eea-kserve-7457f98d7f-vbbxx\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") " pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"
Sep 06 19:32:39.590459 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.590425    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-home\") pod \"noten-model-477eea-kserve-7457f98d7f-vbbxx\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") " pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"
Sep 06 19:32:39.590459 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.590463    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6cb4749-1b55-4a9e-a867-0b51dce53855-tls-certs\") pod \"noten-model-477eea-kserve-7457f98d7f-vbbxx\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") " pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"
Sep 06 19:32:39.590705 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.590493    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-tmp-dir\") pod \"noten-model-477eea-kserve-7457f98d7f-vbbxx\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") " pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"
Sep 06 19:32:39.590705 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.590508    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-dshm\") pod \"noten-model-477eea-kserve-7457f98d7f-vbbxx\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") " pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"
Sep 06 19:32:39.590705 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.590538    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-model-cache\") pod \"noten-model-477eea-kserve-7457f98d7f-vbbxx\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") " pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"
Sep 06 19:32:39.590705 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.590566    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tb8x\" (UniqueName: \"kubernetes.io/projected/e6cb4749-1b55-4a9e-a867-0b51dce53855-kube-api-access-9tb8x\") pod \"noten-model-477eea-kserve-7457f98d7f-vbbxx\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") " pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"
Sep 06 19:32:39.590930 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.590816    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-home\") pod \"noten-model-477eea-kserve-7457f98d7f-vbbxx\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") " pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"
Sep 06 19:32:39.590930 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.590892    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-tmp-dir\") pod \"noten-model-477eea-kserve-7457f98d7f-vbbxx\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") " pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"
Sep 06 19:32:39.591047 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.590964    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-model-cache\") pod \"noten-model-477eea-kserve-7457f98d7f-vbbxx\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") " pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"
Sep 06 19:32:39.592920 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.592901    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-dshm\") pod \"noten-model-477eea-kserve-7457f98d7f-vbbxx\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") " pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"
Sep 06 19:32:39.593165 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.593149    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6cb4749-1b55-4a9e-a867-0b51dce53855-tls-certs\") pod \"noten-model-477eea-kserve-7457f98d7f-vbbxx\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") " pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"
Sep 06 19:32:39.598199 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.598148    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tb8x\" (UniqueName: \"kubernetes.io/projected/e6cb4749-1b55-4a9e-a867-0b51dce53855-kube-api-access-9tb8x\") pod \"noten-model-477eea-kserve-7457f98d7f-vbbxx\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") " pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"
Sep 06 19:32:39.661649 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.661595    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"
Sep 06 19:32:39.932275 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:39.932246    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"]
Sep 06 19:32:39.934212 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:32:39.934183    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6cb4749_1b55_4a9e_a867_0b51dce53855.slice/crio-ff7d793771b506e6a40a9fa33678178a33fb16a515701a5bf9936741aecb09da WatchSource:0}: Error finding container ff7d793771b506e6a40a9fa33678178a33fb16a515701a5bf9936741aecb09da: Status 404 returned error can't find the container with id ff7d793771b506e6a40a9fa33678178a33fb16a515701a5bf9936741aecb09da
Sep 06 19:32:40.475045 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:40.475005    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx" event={"ID":"e6cb4749-1b55-4a9e-a867-0b51dce53855","Type":"ContainerStarted","Data":"5c3cd1d9452ab13e841147901a038508b9fb87e03f5340f9a3a20d57270b06ed"}
Sep 06 19:32:40.475045 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:40.475050    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx" event={"ID":"e6cb4749-1b55-4a9e-a867-0b51dce53855","Type":"ContainerStarted","Data":"ff7d793771b506e6a40a9fa33678178a33fb16a515701a5bf9936741aecb09da"}
Sep 06 19:32:40.493114 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:40.493071    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx" podStartSLOduration=1.493057082 podStartE2EDuration="1.493057082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:32:40.490839253 +0000 UTC m=+1294.672021059" watchObservedRunningTime="2026-09-06 19:32:40.493057082 +0000 UTC m=+1294.674238888"
Sep 06 19:32:43.058830 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:43.058793    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"]
Sep 06 19:32:49.662220 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:49.662183    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"
Sep 06 19:32:53.320724 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:53.320697    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"]
Sep 06 19:32:57.752411 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:57.752377    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:32:58.079252 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.079149    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx" podUID="e6cb4749-1b55-4a9e-a867-0b51dce53855" containerName="main" containerID="cri-o://5c3cd1d9452ab13e841147901a038508b9fb87e03f5340f9a3a20d57270b06ed" gracePeriod=45
Sep 06 19:32:58.361910 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.361888    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"
Sep 06 19:32:58.362763 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.362744    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-tmp-dir\") pod \"e6cb4749-1b55-4a9e-a867-0b51dce53855\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") "
Sep 06 19:32:58.362821 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.362788    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tb8x\" (UniqueName: \"kubernetes.io/projected/e6cb4749-1b55-4a9e-a867-0b51dce53855-kube-api-access-9tb8x\") pod \"e6cb4749-1b55-4a9e-a867-0b51dce53855\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") "
Sep 06 19:32:58.362821 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.362819    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-dshm\") pod \"e6cb4749-1b55-4a9e-a867-0b51dce53855\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") "
Sep 06 19:32:58.362905 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.362865    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6cb4749-1b55-4a9e-a867-0b51dce53855-tls-certs\") pod \"e6cb4749-1b55-4a9e-a867-0b51dce53855\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") "
Sep 06 19:32:58.362905 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.362897    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-model-cache\") pod \"e6cb4749-1b55-4a9e-a867-0b51dce53855\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") "
Sep 06 19:32:58.363020 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.362997    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-home\") pod \"e6cb4749-1b55-4a9e-a867-0b51dce53855\" (UID: \"e6cb4749-1b55-4a9e-a867-0b51dce53855\") "
Sep 06 19:32:58.363101 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.363079    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6cb4749-1b55-4a9e-a867-0b51dce53855" (UID: "e6cb4749-1b55-4a9e-a867-0b51dce53855"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:32:58.363811 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.363274    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-home" (OuterVolumeSpecName: "home") pod "e6cb4749-1b55-4a9e-a867-0b51dce53855" (UID: "e6cb4749-1b55-4a9e-a867-0b51dce53855"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:32:58.363811 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.363525    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-home\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:32:58.363811 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.363546    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-tmp-dir\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:32:58.363811 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.363598    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6cb4749-1b55-4a9e-a867-0b51dce53855" (UID: "e6cb4749-1b55-4a9e-a867-0b51dce53855"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:32:58.365534 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.365501    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6cb4749-1b55-4a9e-a867-0b51dce53855-kube-api-access-9tb8x" (OuterVolumeSpecName: "kube-api-access-9tb8x") pod "e6cb4749-1b55-4a9e-a867-0b51dce53855" (UID: "e6cb4749-1b55-4a9e-a867-0b51dce53855"). InnerVolumeSpecName "kube-api-access-9tb8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:32:58.366243 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.366213    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-dshm" (OuterVolumeSpecName: "dshm") pod "e6cb4749-1b55-4a9e-a867-0b51dce53855" (UID: "e6cb4749-1b55-4a9e-a867-0b51dce53855"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:32:58.366243 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.366213    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6cb4749-1b55-4a9e-a867-0b51dce53855-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6cb4749-1b55-4a9e-a867-0b51dce53855" (UID: "e6cb4749-1b55-4a9e-a867-0b51dce53855"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:32:58.464275 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.464242    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6cb4749-1b55-4a9e-a867-0b51dce53855-tls-certs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:32:58.464275 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.464269    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-model-cache\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:32:58.464275 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.464277    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tb8x\" (UniqueName: \"kubernetes.io/projected/e6cb4749-1b55-4a9e-a867-0b51dce53855-kube-api-access-9tb8x\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:32:58.464478 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.464286    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6cb4749-1b55-4a9e-a867-0b51dce53855-dshm\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:32:58.578703 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.578667    2566 generic.go:356] "Generic (PLEG): container finished" podID="e6cb4749-1b55-4a9e-a867-0b51dce53855" containerID="5c3cd1d9452ab13e841147901a038508b9fb87e03f5340f9a3a20d57270b06ed" exitCode=0
Sep 06 19:32:58.578869 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.578716    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"
Sep 06 19:32:58.578869 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.578735    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx" event={"ID":"e6cb4749-1b55-4a9e-a867-0b51dce53855","Type":"ContainerDied","Data":"5c3cd1d9452ab13e841147901a038508b9fb87e03f5340f9a3a20d57270b06ed"}
Sep 06 19:32:58.578869 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.578777    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx" event={"ID":"e6cb4749-1b55-4a9e-a867-0b51dce53855","Type":"ContainerDied","Data":"ff7d793771b506e6a40a9fa33678178a33fb16a515701a5bf9936741aecb09da"}
Sep 06 19:32:58.578869 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.578801    2566 scope.go:117] "RemoveContainer" containerID="5c3cd1d9452ab13e841147901a038508b9fb87e03f5340f9a3a20d57270b06ed"
Sep 06 19:32:58.589131 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.589112    2566 scope.go:117] "RemoveContainer" containerID="5c3cd1d9452ab13e841147901a038508b9fb87e03f5340f9a3a20d57270b06ed"
Sep 06 19:32:58.589420 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:32:58.589398    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c3cd1d9452ab13e841147901a038508b9fb87e03f5340f9a3a20d57270b06ed\": container with ID starting with 5c3cd1d9452ab13e841147901a038508b9fb87e03f5340f9a3a20d57270b06ed not found: ID does not exist" containerID="5c3cd1d9452ab13e841147901a038508b9fb87e03f5340f9a3a20d57270b06ed"
Sep 06 19:32:58.589474 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.589428    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c3cd1d9452ab13e841147901a038508b9fb87e03f5340f9a3a20d57270b06ed"} err="failed to get container status \"5c3cd1d9452ab13e841147901a038508b9fb87e03f5340f9a3a20d57270b06ed\": rpc error: code = NotFound desc = could not find container \"5c3cd1d9452ab13e841147901a038508b9fb87e03f5340f9a3a20d57270b06ed\": container with ID starting with 5c3cd1d9452ab13e841147901a038508b9fb87e03f5340f9a3a20d57270b06ed not found: ID does not exist"
Sep 06 19:32:58.598933 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.598883    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"]
Sep 06 19:32:58.603126 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:32:58.603101    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-477eea/noten-model-477eea-kserve-7457f98d7f-vbbxx"]
Sep 06 19:33:00.310381 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:00.310340    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6cb4749-1b55-4a9e-a867-0b51dce53855" path="/var/lib/kubelet/pods/e6cb4749-1b55-4a9e-a867-0b51dce53855/volumes"
Sep 06 19:33:03.829491 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:03.829456    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5307a7-29812050-sh24g"]
Sep 06 19:33:03.833413 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:03.833388    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5307a7-29812050-sh24g"]
Sep 06 19:33:03.852300 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:03.852275    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh"]
Sep 06 19:33:03.852552 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:03.852505    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh" podUID="73058c50-af0d-4e86-9a7c-5a897270c5b7" containerName="payload-processing" containerID="cri-o://4b7ddbc0ce5b279d107b1e25e846f505bc895fd79b655b139b321d2131981226" gracePeriod=30
Sep 06 19:33:04.106418 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:04.106396    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh"
Sep 06 19:33:04.220219 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:04.220191    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-794ff\" (UniqueName: \"kubernetes.io/projected/73058c50-af0d-4e86-9a7c-5a897270c5b7-kube-api-access-794ff\") pod \"73058c50-af0d-4e86-9a7c-5a897270c5b7\" (UID: \"73058c50-af0d-4e86-9a7c-5a897270c5b7\") "
Sep 06 19:33:04.220351 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:04.220232    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73058c50-af0d-4e86-9a7c-5a897270c5b7-plugins-config-volume\") pod \"73058c50-af0d-4e86-9a7c-5a897270c5b7\" (UID: \"73058c50-af0d-4e86-9a7c-5a897270c5b7\") "
Sep 06 19:33:04.220624 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:04.220586    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73058c50-af0d-4e86-9a7c-5a897270c5b7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "73058c50-af0d-4e86-9a7c-5a897270c5b7" (UID: "73058c50-af0d-4e86-9a7c-5a897270c5b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:33:04.222389 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:04.222372    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73058c50-af0d-4e86-9a7c-5a897270c5b7-kube-api-access-794ff" (OuterVolumeSpecName: "kube-api-access-794ff") pod "73058c50-af0d-4e86-9a7c-5a897270c5b7" (UID: "73058c50-af0d-4e86-9a7c-5a897270c5b7"). InnerVolumeSpecName "kube-api-access-794ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:33:04.310708 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:04.310684    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8be75dec-2b86-41b5-a254-a83265cd08e6" path="/var/lib/kubelet/pods/8be75dec-2b86-41b5-a254-a83265cd08e6/volumes"
Sep 06 19:33:04.321633 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:04.321597    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-794ff\" (UniqueName: \"kubernetes.io/projected/73058c50-af0d-4e86-9a7c-5a897270c5b7-kube-api-access-794ff\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:33:04.321633 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:04.321632    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73058c50-af0d-4e86-9a7c-5a897270c5b7-plugins-config-volume\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:33:04.616537 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:04.616498    2566 generic.go:356] "Generic (PLEG): container finished" podID="73058c50-af0d-4e86-9a7c-5a897270c5b7" containerID="4b7ddbc0ce5b279d107b1e25e846f505bc895fd79b655b139b321d2131981226" exitCode=0
Sep 06 19:33:04.616744 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:04.616545    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh"
Sep 06 19:33:04.616744 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:04.616566    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh" event={"ID":"73058c50-af0d-4e86-9a7c-5a897270c5b7","Type":"ContainerDied","Data":"4b7ddbc0ce5b279d107b1e25e846f505bc895fd79b655b139b321d2131981226"}
Sep 06 19:33:04.616744 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:04.616630    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh" event={"ID":"73058c50-af0d-4e86-9a7c-5a897270c5b7","Type":"ContainerDied","Data":"a50c011bdad1a93e5c350e50816901a2d0a02a2e81104e9b27d18d6350bb7789"}
Sep 06 19:33:04.616744 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:04.616653    2566 scope.go:117] "RemoveContainer" containerID="4b7ddbc0ce5b279d107b1e25e846f505bc895fd79b655b139b321d2131981226"
Sep 06 19:33:04.626153 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:04.626137    2566 scope.go:117] "RemoveContainer" containerID="4b7ddbc0ce5b279d107b1e25e846f505bc895fd79b655b139b321d2131981226"
Sep 06 19:33:04.626426 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:33:04.626407    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b7ddbc0ce5b279d107b1e25e846f505bc895fd79b655b139b321d2131981226\": container with ID starting with 4b7ddbc0ce5b279d107b1e25e846f505bc895fd79b655b139b321d2131981226 not found: ID does not exist" containerID="4b7ddbc0ce5b279d107b1e25e846f505bc895fd79b655b139b321d2131981226"
Sep 06 19:33:04.626466 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:04.626442    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b7ddbc0ce5b279d107b1e25e846f505bc895fd79b655b139b321d2131981226"} err="failed to get container status \"4b7ddbc0ce5b279d107b1e25e846f505bc895fd79b655b139b321d2131981226\": rpc error: code = NotFound desc = could not find container \"4b7ddbc0ce5b279d107b1e25e846f505bc895fd79b655b139b321d2131981226\": container with ID starting with 4b7ddbc0ce5b279d107b1e25e846f505bc895fd79b655b139b321d2131981226 not found: ID does not exist"
Sep 06 19:33:04.634430 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:04.634398    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh"]
Sep 06 19:33:04.638280 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:04.638255    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5307a7-589554ff47-r8kqh"]
Sep 06 19:33:06.312061 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:06.312027    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73058c50-af0d-4e86-9a7c-5a897270c5b7" path="/var/lib/kubelet/pods/73058c50-af0d-4e86-9a7c-5a897270c5b7/volumes"
Sep 06 19:33:06.350180 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:06.350153    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:33:08.342737 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.342683    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q" podUID="fe80f4f1-77e1-4ffe-8c37-e43b87dbee12" containerName="main" containerID="cri-o://99e0de02b705c51556b8ddbb113b1f142e7338973579838f475a48c4cf7ea9ee" gracePeriod=45
Sep 06 19:33:08.594221 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.594162    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:33:08.639351 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.639322    2566 generic.go:356] "Generic (PLEG): container finished" podID="fe80f4f1-77e1-4ffe-8c37-e43b87dbee12" containerID="99e0de02b705c51556b8ddbb113b1f142e7338973579838f475a48c4cf7ea9ee" exitCode=0
Sep 06 19:33:08.639493 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.639356    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"
Sep 06 19:33:08.639493 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.639407    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q" event={"ID":"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12","Type":"ContainerDied","Data":"99e0de02b705c51556b8ddbb113b1f142e7338973579838f475a48c4cf7ea9ee"}
Sep 06 19:33:08.639493 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.639448    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q" event={"ID":"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12","Type":"ContainerDied","Data":"5e0afe5d0e27a68fccc153ecdab8d0635f51ce4f1bddc08f5fbc8bd9b46a4253"}
Sep 06 19:33:08.639493 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.639466    2566 scope.go:117] "RemoveContainer" containerID="99e0de02b705c51556b8ddbb113b1f142e7338973579838f475a48c4cf7ea9ee"
Sep 06 19:33:08.648285 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.648267    2566 scope.go:117] "RemoveContainer" containerID="99e0de02b705c51556b8ddbb113b1f142e7338973579838f475a48c4cf7ea9ee"
Sep 06 19:33:08.648573 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:33:08.648556    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99e0de02b705c51556b8ddbb113b1f142e7338973579838f475a48c4cf7ea9ee\": container with ID starting with 99e0de02b705c51556b8ddbb113b1f142e7338973579838f475a48c4cf7ea9ee not found: ID does not exist" containerID="99e0de02b705c51556b8ddbb113b1f142e7338973579838f475a48c4cf7ea9ee"
Sep 06 19:33:08.648642 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.648583    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99e0de02b705c51556b8ddbb113b1f142e7338973579838f475a48c4cf7ea9ee"} err="failed to get container status \"99e0de02b705c51556b8ddbb113b1f142e7338973579838f475a48c4cf7ea9ee\": rpc error: code = NotFound desc = could not find container \"99e0de02b705c51556b8ddbb113b1f142e7338973579838f475a48c4cf7ea9ee\": container with ID starting with 99e0de02b705c51556b8ddbb113b1f142e7338973579838f475a48c4cf7ea9ee not found: ID does not exist"
Sep 06 19:33:08.659835 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.659816    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6l7sx\" (UniqueName: \"kubernetes.io/projected/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-kube-api-access-6l7sx\") pod \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") "
Sep 06 19:33:08.659912 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.659877    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-model-cache\") pod \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") "
Sep 06 19:33:08.659953 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.659931    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-home\") pod \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") "
Sep 06 19:33:08.659988 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.659951    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-tmp-dir\") pod \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") "
Sep 06 19:33:08.659988 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.659978    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-dshm\") pod \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") "
Sep 06 19:33:08.660068 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.660014    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-tls-certs\") pod \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\" (UID: \"fe80f4f1-77e1-4ffe-8c37-e43b87dbee12\") "
Sep 06 19:33:08.660191 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.660168    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe80f4f1-77e1-4ffe-8c37-e43b87dbee12" (UID: "fe80f4f1-77e1-4ffe-8c37-e43b87dbee12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:33:08.660191 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.660180    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-home" (OuterVolumeSpecName: "home") pod "fe80f4f1-77e1-4ffe-8c37-e43b87dbee12" (UID: "fe80f4f1-77e1-4ffe-8c37-e43b87dbee12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:33:08.660334 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.660198    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe80f4f1-77e1-4ffe-8c37-e43b87dbee12" (UID: "fe80f4f1-77e1-4ffe-8c37-e43b87dbee12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:33:08.662207 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.662181    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-dshm" (OuterVolumeSpecName: "dshm") pod "fe80f4f1-77e1-4ffe-8c37-e43b87dbee12" (UID: "fe80f4f1-77e1-4ffe-8c37-e43b87dbee12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:33:08.662207 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.662186    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe80f4f1-77e1-4ffe-8c37-e43b87dbee12" (UID: "fe80f4f1-77e1-4ffe-8c37-e43b87dbee12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:33:08.662207 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.662195    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-kube-api-access-6l7sx" (OuterVolumeSpecName: "kube-api-access-6l7sx") pod "fe80f4f1-77e1-4ffe-8c37-e43b87dbee12" (UID: "fe80f4f1-77e1-4ffe-8c37-e43b87dbee12"). InnerVolumeSpecName "kube-api-access-6l7sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:33:08.761336 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.761312    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-tls-certs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:33:08.761336 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.761332    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l7sx\" (UniqueName: \"kubernetes.io/projected/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-kube-api-access-6l7sx\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:33:08.761460 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.761341    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-model-cache\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:33:08.761460 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.761350    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-home\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:33:08.761460 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.761357    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-tmp-dir\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:33:08.761460 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.761365    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12-dshm\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:33:08.961108 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.961037    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"]
Sep 06 19:33:08.963940 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:08.963917    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d73678/test-model-d73678-kserve-6bb9cdbc75-95r6q"]
Sep 06 19:33:10.310454 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:10.310425    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe80f4f1-77e1-4ffe-8c37-e43b87dbee12" path="/var/lib/kubelet/pods/fe80f4f1-77e1-4ffe-8c37-e43b87dbee12/volumes"
Sep 06 19:33:18.148290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.148253    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"]
Sep 06 19:33:18.148915 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.148894    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe80f4f1-77e1-4ffe-8c37-e43b87dbee12" containerName="main"
Sep 06 19:33:18.148968 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.148920    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe80f4f1-77e1-4ffe-8c37-e43b87dbee12" containerName="main"
Sep 06 19:33:18.148968 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.148939    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6cb4749-1b55-4a9e-a867-0b51dce53855" containerName="main"
Sep 06 19:33:18.148968 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.148948    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6cb4749-1b55-4a9e-a867-0b51dce53855" containerName="main"
Sep 06 19:33:18.148968 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.148965    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73058c50-af0d-4e86-9a7c-5a897270c5b7" containerName="payload-processing"
Sep 06 19:33:18.149090 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.148975    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="73058c50-af0d-4e86-9a7c-5a897270c5b7" containerName="payload-processing"
Sep 06 19:33:18.149090 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.149076    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="73058c50-af0d-4e86-9a7c-5a897270c5b7" containerName="payload-processing"
Sep 06 19:33:18.149153 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.149089    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe80f4f1-77e1-4ffe-8c37-e43b87dbee12" containerName="main"
Sep 06 19:33:18.149153 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.149100    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6cb4749-1b55-4a9e-a867-0b51dce53855" containerName="main"
Sep 06 19:33:18.153880 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.153857    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"
Sep 06 19:33:18.156779 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.156754    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-6bbb53-serving-cert\""
Sep 06 19:33:18.160486 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.160397    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"]
Sep 06 19:33:18.246419 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.246389    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f54482c5-1e86-4580-95d5-6e1504bbc177-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr\" (UID: \"f54482c5-1e86-4580-95d5-6e1504bbc177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"
Sep 06 19:33:18.246419 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.246430    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f54482c5-1e86-4580-95d5-6e1504bbc177-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr\" (UID: \"f54482c5-1e86-4580-95d5-6e1504bbc177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"
Sep 06 19:33:18.246741 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.246552    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f54482c5-1e86-4580-95d5-6e1504bbc177-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr\" (UID: \"f54482c5-1e86-4580-95d5-6e1504bbc177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"
Sep 06 19:33:18.246741 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.246647    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5ggs\" (UniqueName: \"kubernetes.io/projected/f54482c5-1e86-4580-95d5-6e1504bbc177-kube-api-access-c5ggs\") pod \"maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr\" (UID: \"f54482c5-1e86-4580-95d5-6e1504bbc177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"
Sep 06 19:33:18.347702 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.347675    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5ggs\" (UniqueName: \"kubernetes.io/projected/f54482c5-1e86-4580-95d5-6e1504bbc177-kube-api-access-c5ggs\") pod \"maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr\" (UID: \"f54482c5-1e86-4580-95d5-6e1504bbc177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"
Sep 06 19:33:18.347874 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.347737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f54482c5-1e86-4580-95d5-6e1504bbc177-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr\" (UID: \"f54482c5-1e86-4580-95d5-6e1504bbc177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"
Sep 06 19:33:18.347874 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.347776    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f54482c5-1e86-4580-95d5-6e1504bbc177-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr\" (UID: \"f54482c5-1e86-4580-95d5-6e1504bbc177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"
Sep 06 19:33:18.347874 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.347841    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f54482c5-1e86-4580-95d5-6e1504bbc177-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr\" (UID: \"f54482c5-1e86-4580-95d5-6e1504bbc177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"
Sep 06 19:33:18.348053 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:33:18.347965    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-serving-cert: secret "maas-api-e2e-shared-b-w2-6bbb53-serving-cert" not found
Sep 06 19:33:18.348053 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:33:18.348044    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f54482c5-1e86-4580-95d5-6e1504bbc177-maas-api-tls podName:f54482c5-1e86-4580-95d5-6e1504bbc177 nodeName:}" failed. No retries permitted until 2026-09-06 19:33:18.848023141 +0000 UTC m=+1333.029204927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f54482c5-1e86-4580-95d5-6e1504bbc177-maas-api-tls") pod "maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr" (UID: "f54482c5-1e86-4580-95d5-6e1504bbc177") : secret "maas-api-e2e-shared-b-w2-6bbb53-serving-cert" not found
Sep 06 19:33:18.348410 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.348388    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f54482c5-1e86-4580-95d5-6e1504bbc177-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr\" (UID: \"f54482c5-1e86-4580-95d5-6e1504bbc177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"
Sep 06 19:33:18.350325 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.350304    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f54482c5-1e86-4580-95d5-6e1504bbc177-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr\" (UID: \"f54482c5-1e86-4580-95d5-6e1504bbc177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"
Sep 06 19:33:18.357544 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.357522    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5ggs\" (UniqueName: \"kubernetes.io/projected/f54482c5-1e86-4580-95d5-6e1504bbc177-kube-api-access-c5ggs\") pod \"maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr\" (UID: \"f54482c5-1e86-4580-95d5-6e1504bbc177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"
Sep 06 19:33:18.852457 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:18.852423    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f54482c5-1e86-4580-95d5-6e1504bbc177-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr\" (UID: \"f54482c5-1e86-4580-95d5-6e1504bbc177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"
Sep 06 19:33:18.852676 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:33:18.852592    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-serving-cert: secret "maas-api-e2e-shared-b-w2-6bbb53-serving-cert" not found
Sep 06 19:33:18.852753 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:33:18.852711    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f54482c5-1e86-4580-95d5-6e1504bbc177-maas-api-tls podName:f54482c5-1e86-4580-95d5-6e1504bbc177 nodeName:}" failed. No retries permitted until 2026-09-06 19:33:19.852687353 +0000 UTC m=+1334.033869151 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f54482c5-1e86-4580-95d5-6e1504bbc177-maas-api-tls") pod "maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr" (UID: "f54482c5-1e86-4580-95d5-6e1504bbc177") : secret "maas-api-e2e-shared-b-w2-6bbb53-serving-cert" not found
Sep 06 19:33:19.862411 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:19.862377    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f54482c5-1e86-4580-95d5-6e1504bbc177-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr\" (UID: \"f54482c5-1e86-4580-95d5-6e1504bbc177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"
Sep 06 19:33:19.864931 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:19.864912    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f54482c5-1e86-4580-95d5-6e1504bbc177-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr\" (UID: \"f54482c5-1e86-4580-95d5-6e1504bbc177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"
Sep 06 19:33:19.968257 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:19.968229    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"
Sep 06 19:33:20.239969 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:20.239946    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"]
Sep 06 19:33:20.241694 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:33:20.241665    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54482c5_1e86_4580_95d5_6e1504bbc177.slice/crio-cd29310cc9151dce5a885bee46339fb15a6fe588276154592a258ce82a5889fd WatchSource:0}: Error finding container cd29310cc9151dce5a885bee46339fb15a6fe588276154592a258ce82a5889fd: Status 404 returned error can't find the container with id cd29310cc9151dce5a885bee46339fb15a6fe588276154592a258ce82a5889fd
Sep 06 19:33:20.243259 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:20.243236    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 19:33:20.707287 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:20.707254    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr" event={"ID":"f54482c5-1e86-4580-95d5-6e1504bbc177","Type":"ContainerStarted","Data":"3b2e0fb682386203956b8fd92654b5c05c7bc990fd8817546d26c9c6d717a774"}
Sep 06 19:33:20.707431 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:20.707302    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr" event={"ID":"f54482c5-1e86-4580-95d5-6e1504bbc177","Type":"ContainerStarted","Data":"cd29310cc9151dce5a885bee46339fb15a6fe588276154592a258ce82a5889fd"}
Sep 06 19:33:20.707431 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:20.707389    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"
Sep 06 19:33:20.724458 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:20.724413    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr" podStartSLOduration=2.372744612 podStartE2EDuration="2.724399726s" totalImagesPullingTime="351.655114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:33:20.722892721 +0000 UTC m=+1334.904074526" watchObservedRunningTime="2026-09-06 19:33:20.724399726 +0000 UTC m=+1334.905581552"
Sep 06 19:33:21.168186 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.168158    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"]
Sep 06 19:33:21.168627 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.168355    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j" podUID="c493a985-9a3a-45cb-a5a3-db914fde1e6b" containerName="maas-api" containerID="cri-o://c7127f8f1d0a7cd32cc90dabd7f7cd5166ec6cb42244c4a602a5c0ddef58187e" gracePeriod=30
Sep 06 19:33:21.199565 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.199532    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d73678-29812050-mp227"]
Sep 06 19:33:21.201629 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.201595    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d73678-29812050-mp227"]
Sep 06 19:33:21.221571 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.221548    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf"]
Sep 06 19:33:21.221796 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.221772    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf" podUID="33e078b2-1619-4b07-90c6-17cb2fe2edf1" containerName="payload-processing" containerID="cri-o://c2a862d682748bd8e5ce4fc0457b4c70664f7b26aaff9f6b180332edee5be4d1" gracePeriod=30
Sep 06 19:33:21.233666 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.233640    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x"]
Sep 06 19:33:21.233865 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.233844    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x" podUID="b5f68d30-5da0-450e-b18c-ddf97c6e4028" containerName="payload-pre-processing" containerID="cri-o://1adfb56e22c3a2ebd85a5d646c37a09d23c4d155671a40b3a9f6ac51c31759ff" gracePeriod=30
Sep 06 19:33:21.525974 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.525939    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf"
Sep 06 19:33:21.529162 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.529141    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x"
Sep 06 19:33:21.616497 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.616473    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"
Sep 06 19:33:21.680416 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.680342    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c493a985-9a3a-45cb-a5a3-db914fde1e6b-maas-api-tls\") pod \"c493a985-9a3a-45cb-a5a3-db914fde1e6b\" (UID: \"c493a985-9a3a-45cb-a5a3-db914fde1e6b\") "
Sep 06 19:33:21.680416 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.680399    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c493a985-9a3a-45cb-a5a3-db914fde1e6b-openshift-service-ca\") pod \"c493a985-9a3a-45cb-a5a3-db914fde1e6b\" (UID: \"c493a985-9a3a-45cb-a5a3-db914fde1e6b\") "
Sep 06 19:33:21.680637 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.680433    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c493a985-9a3a-45cb-a5a3-db914fde1e6b-metrics-certs\") pod \"c493a985-9a3a-45cb-a5a3-db914fde1e6b\" (UID: \"c493a985-9a3a-45cb-a5a3-db914fde1e6b\") "
Sep 06 19:33:21.680637 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.680456    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f68d30-5da0-450e-b18c-ddf97c6e4028-plugins-config-volume\") pod \"b5f68d30-5da0-450e-b18c-ddf97c6e4028\" (UID: \"b5f68d30-5da0-450e-b18c-ddf97c6e4028\") "
Sep 06 19:33:21.680637 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.680519    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72t6r\" (UniqueName: \"kubernetes.io/projected/33e078b2-1619-4b07-90c6-17cb2fe2edf1-kube-api-access-72t6r\") pod \"33e078b2-1619-4b07-90c6-17cb2fe2edf1\" (UID: \"33e078b2-1619-4b07-90c6-17cb2fe2edf1\") "
Sep 06 19:33:21.680637 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.680557    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33e078b2-1619-4b07-90c6-17cb2fe2edf1-plugins-config-volume\") pod \"33e078b2-1619-4b07-90c6-17cb2fe2edf1\" (UID: \"33e078b2-1619-4b07-90c6-17cb2fe2edf1\") "
Sep 06 19:33:21.680846 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.680678    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nd4p\" (UniqueName: \"kubernetes.io/projected/c493a985-9a3a-45cb-a5a3-db914fde1e6b-kube-api-access-9nd4p\") pod \"c493a985-9a3a-45cb-a5a3-db914fde1e6b\" (UID: \"c493a985-9a3a-45cb-a5a3-db914fde1e6b\") "
Sep 06 19:33:21.680846 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.680704    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srtzq\" (UniqueName: \"kubernetes.io/projected/b5f68d30-5da0-450e-b18c-ddf97c6e4028-kube-api-access-srtzq\") pod \"b5f68d30-5da0-450e-b18c-ddf97c6e4028\" (UID: \"b5f68d30-5da0-450e-b18c-ddf97c6e4028\") "
Sep 06 19:33:21.680846 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.680819    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c493a985-9a3a-45cb-a5a3-db914fde1e6b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c493a985-9a3a-45cb-a5a3-db914fde1e6b" (UID: "c493a985-9a3a-45cb-a5a3-db914fde1e6b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:33:21.680987 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.680911    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f68d30-5da0-450e-b18c-ddf97c6e4028-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b5f68d30-5da0-450e-b18c-ddf97c6e4028" (UID: "b5f68d30-5da0-450e-b18c-ddf97c6e4028"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:33:21.681046 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.681009    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c493a985-9a3a-45cb-a5a3-db914fde1e6b-openshift-service-ca\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:33:21.681046 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.681028    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f68d30-5da0-450e-b18c-ddf97c6e4028-plugins-config-volume\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:33:21.681270 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.681201    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e078b2-1619-4b07-90c6-17cb2fe2edf1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "33e078b2-1619-4b07-90c6-17cb2fe2edf1" (UID: "33e078b2-1619-4b07-90c6-17cb2fe2edf1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:33:21.682979 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.682951    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c493a985-9a3a-45cb-a5a3-db914fde1e6b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c493a985-9a3a-45cb-a5a3-db914fde1e6b" (UID: "c493a985-9a3a-45cb-a5a3-db914fde1e6b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:33:21.682979 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.682968    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c493a985-9a3a-45cb-a5a3-db914fde1e6b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c493a985-9a3a-45cb-a5a3-db914fde1e6b" (UID: "c493a985-9a3a-45cb-a5a3-db914fde1e6b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:33:21.683134 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.683051    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f68d30-5da0-450e-b18c-ddf97c6e4028-kube-api-access-srtzq" (OuterVolumeSpecName: "kube-api-access-srtzq") pod "b5f68d30-5da0-450e-b18c-ddf97c6e4028" (UID: "b5f68d30-5da0-450e-b18c-ddf97c6e4028"). InnerVolumeSpecName "kube-api-access-srtzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:33:21.683386 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.683363    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c493a985-9a3a-45cb-a5a3-db914fde1e6b-kube-api-access-9nd4p" (OuterVolumeSpecName: "kube-api-access-9nd4p") pod "c493a985-9a3a-45cb-a5a3-db914fde1e6b" (UID: "c493a985-9a3a-45cb-a5a3-db914fde1e6b"). InnerVolumeSpecName "kube-api-access-9nd4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:33:21.683450 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.683395    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e078b2-1619-4b07-90c6-17cb2fe2edf1-kube-api-access-72t6r" (OuterVolumeSpecName: "kube-api-access-72t6r") pod "33e078b2-1619-4b07-90c6-17cb2fe2edf1" (UID: "33e078b2-1619-4b07-90c6-17cb2fe2edf1"). InnerVolumeSpecName "kube-api-access-72t6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:33:21.713768 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.713744    2566 generic.go:356] "Generic (PLEG): container finished" podID="33e078b2-1619-4b07-90c6-17cb2fe2edf1" containerID="c2a862d682748bd8e5ce4fc0457b4c70664f7b26aaff9f6b180332edee5be4d1" exitCode=0
Sep 06 19:33:21.713882 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.713796    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf"
Sep 06 19:33:21.713882 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.713833    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf" event={"ID":"33e078b2-1619-4b07-90c6-17cb2fe2edf1","Type":"ContainerDied","Data":"c2a862d682748bd8e5ce4fc0457b4c70664f7b26aaff9f6b180332edee5be4d1"}
Sep 06 19:33:21.713998 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.713922    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf" event={"ID":"33e078b2-1619-4b07-90c6-17cb2fe2edf1","Type":"ContainerDied","Data":"b10d2963d3225f70678d824e86cd136dac5f53f0f879c70b163d79304b55363a"}
Sep 06 19:33:21.713998 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.713956    2566 scope.go:117] "RemoveContainer" containerID="c2a862d682748bd8e5ce4fc0457b4c70664f7b26aaff9f6b180332edee5be4d1"
Sep 06 19:33:21.715194 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.715174    2566 generic.go:356] "Generic (PLEG): container finished" podID="b5f68d30-5da0-450e-b18c-ddf97c6e4028" containerID="1adfb56e22c3a2ebd85a5d646c37a09d23c4d155671a40b3a9f6ac51c31759ff" exitCode=0
Sep 06 19:33:21.715278 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.715230    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x"
Sep 06 19:33:21.715278 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.715253    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x" event={"ID":"b5f68d30-5da0-450e-b18c-ddf97c6e4028","Type":"ContainerDied","Data":"1adfb56e22c3a2ebd85a5d646c37a09d23c4d155671a40b3a9f6ac51c31759ff"}
Sep 06 19:33:21.715551 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.715298    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x" event={"ID":"b5f68d30-5da0-450e-b18c-ddf97c6e4028","Type":"ContainerDied","Data":"d13c947533b1dda8a243729f2c128ede302d5250a0ab62ddc3109b1e9c9be621"}
Sep 06 19:33:21.716873 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.716852    2566 generic.go:356] "Generic (PLEG): container finished" podID="c493a985-9a3a-45cb-a5a3-db914fde1e6b" containerID="c7127f8f1d0a7cd32cc90dabd7f7cd5166ec6cb42244c4a602a5c0ddef58187e" exitCode=0
Sep 06 19:33:21.717006 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.716881    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j" event={"ID":"c493a985-9a3a-45cb-a5a3-db914fde1e6b","Type":"ContainerDied","Data":"c7127f8f1d0a7cd32cc90dabd7f7cd5166ec6cb42244c4a602a5c0ddef58187e"}
Sep 06 19:33:21.717006 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.716917    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"
Sep 06 19:33:21.717006 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.716928    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j" event={"ID":"c493a985-9a3a-45cb-a5a3-db914fde1e6b","Type":"ContainerDied","Data":"8037042283c412c51c654feb08313209ce1a495c116715ac32c1af49319032c2"}
Sep 06 19:33:21.724391 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.724094    2566 scope.go:117] "RemoveContainer" containerID="c2a862d682748bd8e5ce4fc0457b4c70664f7b26aaff9f6b180332edee5be4d1"
Sep 06 19:33:21.724550 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:33:21.724430    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a862d682748bd8e5ce4fc0457b4c70664f7b26aaff9f6b180332edee5be4d1\": container with ID starting with c2a862d682748bd8e5ce4fc0457b4c70664f7b26aaff9f6b180332edee5be4d1 not found: ID does not exist" containerID="c2a862d682748bd8e5ce4fc0457b4c70664f7b26aaff9f6b180332edee5be4d1"
Sep 06 19:33:21.724550 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.724460    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a862d682748bd8e5ce4fc0457b4c70664f7b26aaff9f6b180332edee5be4d1"} err="failed to get container status \"c2a862d682748bd8e5ce4fc0457b4c70664f7b26aaff9f6b180332edee5be4d1\": rpc error: code = NotFound desc = could not find container \"c2a862d682748bd8e5ce4fc0457b4c70664f7b26aaff9f6b180332edee5be4d1\": container with ID starting with c2a862d682748bd8e5ce4fc0457b4c70664f7b26aaff9f6b180332edee5be4d1 not found: ID does not exist"
Sep 06 19:33:21.724550 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.724480    2566 scope.go:117] "RemoveContainer" containerID="1adfb56e22c3a2ebd85a5d646c37a09d23c4d155671a40b3a9f6ac51c31759ff"
Sep 06 19:33:21.734515 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.734497    2566 scope.go:117] "RemoveContainer" containerID="1adfb56e22c3a2ebd85a5d646c37a09d23c4d155671a40b3a9f6ac51c31759ff"
Sep 06 19:33:21.734806 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:33:21.734790    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1adfb56e22c3a2ebd85a5d646c37a09d23c4d155671a40b3a9f6ac51c31759ff\": container with ID starting with 1adfb56e22c3a2ebd85a5d646c37a09d23c4d155671a40b3a9f6ac51c31759ff not found: ID does not exist" containerID="1adfb56e22c3a2ebd85a5d646c37a09d23c4d155671a40b3a9f6ac51c31759ff"
Sep 06 19:33:21.734857 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.734813    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1adfb56e22c3a2ebd85a5d646c37a09d23c4d155671a40b3a9f6ac51c31759ff"} err="failed to get container status \"1adfb56e22c3a2ebd85a5d646c37a09d23c4d155671a40b3a9f6ac51c31759ff\": rpc error: code = NotFound desc = could not find container \"1adfb56e22c3a2ebd85a5d646c37a09d23c4d155671a40b3a9f6ac51c31759ff\": container with ID starting with 1adfb56e22c3a2ebd85a5d646c37a09d23c4d155671a40b3a9f6ac51c31759ff not found: ID does not exist"
Sep 06 19:33:21.734857 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.734830    2566 scope.go:117] "RemoveContainer" containerID="c7127f8f1d0a7cd32cc90dabd7f7cd5166ec6cb42244c4a602a5c0ddef58187e"
Sep 06 19:33:21.740240 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.740214    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf"]
Sep 06 19:33:21.744009 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.743984    2566 scope.go:117] "RemoveContainer" containerID="c7127f8f1d0a7cd32cc90dabd7f7cd5166ec6cb42244c4a602a5c0ddef58187e"
Sep 06 19:33:21.744277 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:33:21.744258    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7127f8f1d0a7cd32cc90dabd7f7cd5166ec6cb42244c4a602a5c0ddef58187e\": container with ID starting with c7127f8f1d0a7cd32cc90dabd7f7cd5166ec6cb42244c4a602a5c0ddef58187e not found: ID does not exist" containerID="c7127f8f1d0a7cd32cc90dabd7f7cd5166ec6cb42244c4a602a5c0ddef58187e"
Sep 06 19:33:21.744339 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.744283    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7127f8f1d0a7cd32cc90dabd7f7cd5166ec6cb42244c4a602a5c0ddef58187e"} err="failed to get container status \"c7127f8f1d0a7cd32cc90dabd7f7cd5166ec6cb42244c4a602a5c0ddef58187e\": rpc error: code = NotFound desc = could not find container \"c7127f8f1d0a7cd32cc90dabd7f7cd5166ec6cb42244c4a602a5c0ddef58187e\": container with ID starting with c7127f8f1d0a7cd32cc90dabd7f7cd5166ec6cb42244c4a602a5c0ddef58187e not found: ID does not exist"
Sep 06 19:33:21.744968 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.744952    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d73678-67c9557b8f-lgbmf"]
Sep 06 19:33:21.755155 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.755136    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"]
Sep 06 19:33:21.757133 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.757115    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d73678-847cbc69f-szf5j"]
Sep 06 19:33:21.766265 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.766246    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x"]
Sep 06 19:33:21.769858 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.769836    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d73678-7bc8dd9658-x222x"]
Sep 06 19:33:21.782476 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.782450    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72t6r\" (UniqueName: \"kubernetes.io/projected/33e078b2-1619-4b07-90c6-17cb2fe2edf1-kube-api-access-72t6r\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:33:21.782476 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.782471    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33e078b2-1619-4b07-90c6-17cb2fe2edf1-plugins-config-volume\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:33:21.782649 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.782487    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nd4p\" (UniqueName: \"kubernetes.io/projected/c493a985-9a3a-45cb-a5a3-db914fde1e6b-kube-api-access-9nd4p\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:33:21.782649 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.782501    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srtzq\" (UniqueName: \"kubernetes.io/projected/b5f68d30-5da0-450e-b18c-ddf97c6e4028-kube-api-access-srtzq\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:33:21.782649 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.782514    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c493a985-9a3a-45cb-a5a3-db914fde1e6b-maas-api-tls\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:33:21.782649 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:21.782527    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c493a985-9a3a-45cb-a5a3-db914fde1e6b-metrics-certs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:33:22.310587 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:22.310558    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33e078b2-1619-4b07-90c6-17cb2fe2edf1" path="/var/lib/kubelet/pods/33e078b2-1619-4b07-90c6-17cb2fe2edf1/volumes"
Sep 06 19:33:22.310942 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:22.310848    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaa44ab2-ee5e-4e29-b873-e908fd6de7f3" path="/var/lib/kubelet/pods/aaa44ab2-ee5e-4e29-b873-e908fd6de7f3/volumes"
Sep 06 19:33:22.311108 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:22.311096    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5f68d30-5da0-450e-b18c-ddf97c6e4028" path="/var/lib/kubelet/pods/b5f68d30-5da0-450e-b18c-ddf97c6e4028/volumes"
Sep 06 19:33:22.311436 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:22.311425    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c493a985-9a3a-45cb-a5a3-db914fde1e6b" path="/var/lib/kubelet/pods/c493a985-9a3a-45cb-a5a3-db914fde1e6b/volumes"
Sep 06 19:33:23.448866 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:23.448836    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:33:26.722549 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:26.722512    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"
Sep 06 19:33:32.216843 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.216755    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"]
Sep 06 19:33:32.217296 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.217160    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c493a985-9a3a-45cb-a5a3-db914fde1e6b" containerName="maas-api"
Sep 06 19:33:32.217296 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.217176    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c493a985-9a3a-45cb-a5a3-db914fde1e6b" containerName="maas-api"
Sep 06 19:33:32.217296 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.217197    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5f68d30-5da0-450e-b18c-ddf97c6e4028" containerName="payload-pre-processing"
Sep 06 19:33:32.217296 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.217203    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5f68d30-5da0-450e-b18c-ddf97c6e4028" containerName="payload-pre-processing"
Sep 06 19:33:32.217296 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.217208    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33e078b2-1619-4b07-90c6-17cb2fe2edf1" containerName="payload-processing"
Sep 06 19:33:32.217296 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.217213    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="33e078b2-1619-4b07-90c6-17cb2fe2edf1" containerName="payload-processing"
Sep 06 19:33:32.217296 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.217278    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5f68d30-5da0-450e-b18c-ddf97c6e4028" containerName="payload-pre-processing"
Sep 06 19:33:32.217296 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.217286    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="33e078b2-1619-4b07-90c6-17cb2fe2edf1" containerName="payload-processing"
Sep 06 19:33:32.217296 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.217292    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c493a985-9a3a-45cb-a5a3-db914fde1e6b" containerName="maas-api"
Sep 06 19:33:32.222015 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.221995    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"
Sep 06 19:33:32.224803 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.224778    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-e5ef25-serving-cert\""
Sep 06 19:33:32.231857 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.231829    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"]
Sep 06 19:33:32.378903 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.378870    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr\" (UID: \"355f0e9b-89be-4768-ab1a-4d6ca4b56b72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"
Sep 06 19:33:32.379090 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.378939    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwxsg\" (UniqueName: \"kubernetes.io/projected/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-kube-api-access-dwxsg\") pod \"maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr\" (UID: \"355f0e9b-89be-4768-ab1a-4d6ca4b56b72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"
Sep 06 19:33:32.379090 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.378978    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr\" (UID: \"355f0e9b-89be-4768-ab1a-4d6ca4b56b72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"
Sep 06 19:33:32.379174 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.379088    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr\" (UID: \"355f0e9b-89be-4768-ab1a-4d6ca4b56b72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"
Sep 06 19:33:32.480061 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.479959    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr\" (UID: \"355f0e9b-89be-4768-ab1a-4d6ca4b56b72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"
Sep 06 19:33:32.480061 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.480022    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr\" (UID: \"355f0e9b-89be-4768-ab1a-4d6ca4b56b72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"
Sep 06 19:33:32.480061 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.480063    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwxsg\" (UniqueName: \"kubernetes.io/projected/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-kube-api-access-dwxsg\") pod \"maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr\" (UID: \"355f0e9b-89be-4768-ab1a-4d6ca4b56b72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"
Sep 06 19:33:32.480339 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.480090    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr\" (UID: \"355f0e9b-89be-4768-ab1a-4d6ca4b56b72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"
Sep 06 19:33:32.480766 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.480748    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr\" (UID: \"355f0e9b-89be-4768-ab1a-4d6ca4b56b72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"
Sep 06 19:33:32.482432 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.482404    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr\" (UID: \"355f0e9b-89be-4768-ab1a-4d6ca4b56b72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"
Sep 06 19:33:32.482524 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.482460    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr\" (UID: \"355f0e9b-89be-4768-ab1a-4d6ca4b56b72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"
Sep 06 19:33:32.489885 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.489855    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwxsg\" (UniqueName: \"kubernetes.io/projected/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-kube-api-access-dwxsg\") pod \"maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr\" (UID: \"355f0e9b-89be-4768-ab1a-4d6ca4b56b72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"
Sep 06 19:33:32.535700 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.535673    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"
Sep 06 19:33:32.829357 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.829329    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"]
Sep 06 19:33:32.830339 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:33:32.830310    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355f0e9b_89be_4768_ab1a_4d6ca4b56b72.slice/crio-4c1848257481279e8db8a60425c094a88bd2e81f11bd0678f6ee7c7f72910ca8 WatchSource:0}: Error finding container 4c1848257481279e8db8a60425c094a88bd2e81f11bd0678f6ee7c7f72910ca8: Status 404 returned error can't find the container with id 4c1848257481279e8db8a60425c094a88bd2e81f11bd0678f6ee7c7f72910ca8
Sep 06 19:33:32.873794 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.873765    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t"]
Sep 06 19:33:32.878347 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.878328    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t"
Sep 06 19:33:32.881471 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.881439    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-e5ef25\""
Sep 06 19:33:32.901857 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.901835    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t"]
Sep 06 19:33:32.931358 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.931336    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn"]
Sep 06 19:33:32.934696 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.934678    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn"
Sep 06 19:33:32.937215 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.937194    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-e5ef25-dockercfg-nsq4j\""
Sep 06 19:33:32.945765 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.945743    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn"]
Sep 06 19:33:32.983929 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.983896    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0126ebcf-90d4-41e5-a692-d2543e6441d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t\" (UID: \"0126ebcf-90d4-41e5-a692-d2543e6441d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t"
Sep 06 19:33:32.984057 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:32.983963    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptx2v\" (UniqueName: \"kubernetes.io/projected/0126ebcf-90d4-41e5-a692-d2543e6441d2-kube-api-access-ptx2v\") pod \"payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t\" (UID: \"0126ebcf-90d4-41e5-a692-d2543e6441d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t"
Sep 06 19:33:33.084549 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.084476    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptx2v\" (UniqueName: \"kubernetes.io/projected/0126ebcf-90d4-41e5-a692-d2543e6441d2-kube-api-access-ptx2v\") pod \"payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t\" (UID: \"0126ebcf-90d4-41e5-a692-d2543e6441d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t"
Sep 06 19:33:33.084549 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.084546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0126ebcf-90d4-41e5-a692-d2543e6441d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t\" (UID: \"0126ebcf-90d4-41e5-a692-d2543e6441d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t"
Sep 06 19:33:33.084775 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.084568    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0a4b1ad-2580-49e5-b64f-a5080137aa8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn\" (UID: \"a0a4b1ad-2580-49e5-b64f-a5080137aa8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn"
Sep 06 19:33:33.084775 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.084624    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9mg9\" (UniqueName: \"kubernetes.io/projected/a0a4b1ad-2580-49e5-b64f-a5080137aa8f-kube-api-access-s9mg9\") pod \"payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn\" (UID: \"a0a4b1ad-2580-49e5-b64f-a5080137aa8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn"
Sep 06 19:33:33.085185 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.085166    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0126ebcf-90d4-41e5-a692-d2543e6441d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t\" (UID: \"0126ebcf-90d4-41e5-a692-d2543e6441d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t"
Sep 06 19:33:33.092841 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.092820    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptx2v\" (UniqueName: \"kubernetes.io/projected/0126ebcf-90d4-41e5-a692-d2543e6441d2-kube-api-access-ptx2v\") pod \"payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t\" (UID: \"0126ebcf-90d4-41e5-a692-d2543e6441d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t"
Sep 06 19:33:33.185721 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.185684    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0a4b1ad-2580-49e5-b64f-a5080137aa8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn\" (UID: \"a0a4b1ad-2580-49e5-b64f-a5080137aa8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn"
Sep 06 19:33:33.185873 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.185735    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9mg9\" (UniqueName: \"kubernetes.io/projected/a0a4b1ad-2580-49e5-b64f-a5080137aa8f-kube-api-access-s9mg9\") pod \"payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn\" (UID: \"a0a4b1ad-2580-49e5-b64f-a5080137aa8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn"
Sep 06 19:33:33.186370 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.186351    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0a4b1ad-2580-49e5-b64f-a5080137aa8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn\" (UID: \"a0a4b1ad-2580-49e5-b64f-a5080137aa8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn"
Sep 06 19:33:33.190795 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.190777    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t"
Sep 06 19:33:33.193200 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.193178    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9mg9\" (UniqueName: \"kubernetes.io/projected/a0a4b1ad-2580-49e5-b64f-a5080137aa8f-kube-api-access-s9mg9\") pod \"payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn\" (UID: \"a0a4b1ad-2580-49e5-b64f-a5080137aa8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn"
Sep 06 19:33:33.245052 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.244868    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn"
Sep 06 19:33:33.463905 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.463882    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t"]
Sep 06 19:33:33.465507 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:33:33.465483    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0126ebcf_90d4_41e5_a692_d2543e6441d2.slice/crio-5c63488b095218f5d61aa956f4b325f33f66cdaaf116b28d00e56f10b0ecb591 WatchSource:0}: Error finding container 5c63488b095218f5d61aa956f4b325f33f66cdaaf116b28d00e56f10b0ecb591: Status 404 returned error can't find the container with id 5c63488b095218f5d61aa956f4b325f33f66cdaaf116b28d00e56f10b0ecb591
Sep 06 19:33:33.529378 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.529354    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn"]
Sep 06 19:33:33.539827 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:33:33.539801    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a4b1ad_2580_49e5_b64f_a5080137aa8f.slice/crio-fcb8bb6735e0514e787a0f32e4bc9dd010f01c144edf21018102c703c4d7f66c WatchSource:0}: Error finding container fcb8bb6735e0514e787a0f32e4bc9dd010f01c144edf21018102c703c4d7f66c: Status 404 returned error can't find the container with id fcb8bb6735e0514e787a0f32e4bc9dd010f01c144edf21018102c703c4d7f66c
Sep 06 19:33:33.787857 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.787763    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn" event={"ID":"a0a4b1ad-2580-49e5-b64f-a5080137aa8f","Type":"ContainerStarted","Data":"e1c594a3be33d035fd48ed83093788522e797edc7049bc01814e4331745e79eb"}
Sep 06 19:33:33.787857 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.787823    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn" event={"ID":"a0a4b1ad-2580-49e5-b64f-a5080137aa8f","Type":"ContainerStarted","Data":"fcb8bb6735e0514e787a0f32e4bc9dd010f01c144edf21018102c703c4d7f66c"}
Sep 06 19:33:33.788086 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.787864    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn"
Sep 06 19:33:33.789842 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.789816    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t" event={"ID":"0126ebcf-90d4-41e5-a692-d2543e6441d2","Type":"ContainerStarted","Data":"35a517016ae406e4e4e972ebcb14ed4c4831053e2c7e7abe9ae8f34b7020cc4c"}
Sep 06 19:33:33.789969 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.789864    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t" event={"ID":"0126ebcf-90d4-41e5-a692-d2543e6441d2","Type":"ContainerStarted","Data":"5c63488b095218f5d61aa956f4b325f33f66cdaaf116b28d00e56f10b0ecb591"}
Sep 06 19:33:33.789969 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.789895    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t"
Sep 06 19:33:33.791523 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.791492    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr" event={"ID":"355f0e9b-89be-4768-ab1a-4d6ca4b56b72","Type":"ContainerStarted","Data":"f9c823f6c47b5d11ace029f1ea94410e3a4c838afda959102163e9172eb9376b"}
Sep 06 19:33:33.791659 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.791551    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"
Sep 06 19:33:33.791659 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.791572    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr" event={"ID":"355f0e9b-89be-4768-ab1a-4d6ca4b56b72","Type":"ContainerStarted","Data":"4c1848257481279e8db8a60425c094a88bd2e81f11bd0678f6ee7c7f72910ca8"}
Sep 06 19:33:33.802889 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.802849    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn" podStartSLOduration=1.802835626 podStartE2EDuration="1.802835626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:33:33.802423288 +0000 UTC m=+1347.983605095" watchObservedRunningTime="2026-09-06 19:33:33.802835626 +0000 UTC m=+1347.984017477"
Sep 06 19:33:33.819542 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.819494    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr" podStartSLOduration=1.408507922 podStartE2EDuration="1.819481598s" totalImagesPullingTime="410.973676ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:33:33.817100971 +0000 UTC m=+1347.998282778" watchObservedRunningTime="2026-09-06 19:33:33.819481598 +0000 UTC m=+1348.000663405"
Sep 06 19:33:33.833196 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:33.833155    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t" podStartSLOduration=1.8331433289999999 podStartE2EDuration="1.833143329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:33:33.831510432 +0000 UTC m=+1348.012692238" watchObservedRunningTime="2026-09-06 19:33:33.833143329 +0000 UTC m=+1348.014325155"
Sep 06 19:33:36.757906 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:36.757877    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:33:39.802802 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:39.802770    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"
Sep 06 19:33:42.568247 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:42.568209    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7719-140153893983552"]
Sep 06 19:33:42.573396 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:42.573365    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-140153893983552"
Sep 06 19:33:42.575871 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:42.575847    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zsskm\""
Sep 06 19:33:42.576218 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:42.576198    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7719-140153893983552"]
Sep 06 19:33:42.672825 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:42.672789    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdjbp\" (UniqueName: \"kubernetes.io/projected/19f69a9f-6923-479f-823c-48994c31fd89-kube-api-access-kdjbp\") pod \"test-curl-7719-140153893983552\" (UID: \"19f69a9f-6923-479f-823c-48994c31fd89\") " pod="opendatahub/test-curl-7719-140153893983552"
Sep 06 19:33:42.773257 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:42.773220    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdjbp\" (UniqueName: \"kubernetes.io/projected/19f69a9f-6923-479f-823c-48994c31fd89-kube-api-access-kdjbp\") pod \"test-curl-7719-140153893983552\" (UID: \"19f69a9f-6923-479f-823c-48994c31fd89\") " pod="opendatahub/test-curl-7719-140153893983552"
Sep 06 19:33:42.782145 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:42.782121    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdjbp\" (UniqueName: \"kubernetes.io/projected/19f69a9f-6923-479f-823c-48994c31fd89-kube-api-access-kdjbp\") pod \"test-curl-7719-140153893983552\" (UID: \"19f69a9f-6923-479f-823c-48994c31fd89\") " pod="opendatahub/test-curl-7719-140153893983552"
Sep 06 19:33:42.883457 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:42.883381    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-140153893983552"
Sep 06 19:33:43.152362 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:43.152342    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7719-140153893983552"]
Sep 06 19:33:43.154246 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:33:43.154218    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19f69a9f_6923_479f_823c_48994c31fd89.slice/crio-575492e49d705af4589c5fba026da3a60a0db853210972142e879ab209311c33 WatchSource:0}: Error finding container 575492e49d705af4589c5fba026da3a60a0db853210972142e879ab209311c33: Status 404 returned error can't find the container with id 575492e49d705af4589c5fba026da3a60a0db853210972142e879ab209311c33
Sep 06 19:33:43.853980 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:43.853942    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7719-140153893983552" event={"ID":"19f69a9f-6923-479f-823c-48994c31fd89","Type":"ContainerStarted","Data":"575492e49d705af4589c5fba026da3a60a0db853210972142e879ab209311c33"}
Sep 06 19:33:44.800335 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:44.800306    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t"
Sep 06 19:33:44.800544 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:44.800379    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn"
Sep 06 19:33:44.861069 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:44.861035    2566 generic.go:356] "Generic (PLEG): container finished" podID="19f69a9f-6923-479f-823c-48994c31fd89" containerID="31462b444f7cf237f643079e6c7de89e4703e31b39d03dce52f369226891b4d2" exitCode=0
Sep 06 19:33:44.861428 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:44.861108    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7719-140153893983552" event={"ID":"19f69a9f-6923-479f-823c-48994c31fd89","Type":"ContainerDied","Data":"31462b444f7cf237f643079e6c7de89e4703e31b39d03dce52f369226891b4d2"}
Sep 06 19:33:45.995944 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:45.995923    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-140153893983552"
Sep 06 19:33:46.027355 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:46.027332    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7719-140153893983552_19f69a9f-6923-479f-823c-48994c31fd89/test-curl-7719-140153893983552/0.log"
Sep 06 19:33:46.048395 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:46.048363    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7719-140153893983552"]
Sep 06 19:33:46.050644 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:46.050627    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7719-140153893983552"]
Sep 06 19:33:46.100268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:46.100233    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdjbp\" (UniqueName: \"kubernetes.io/projected/19f69a9f-6923-479f-823c-48994c31fd89-kube-api-access-kdjbp\") pod \"19f69a9f-6923-479f-823c-48994c31fd89\" (UID: \"19f69a9f-6923-479f-823c-48994c31fd89\") "
Sep 06 19:33:46.102335 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:46.102309    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f69a9f-6923-479f-823c-48994c31fd89-kube-api-access-kdjbp" (OuterVolumeSpecName: "kube-api-access-kdjbp") pod "19f69a9f-6923-479f-823c-48994c31fd89" (UID: "19f69a9f-6923-479f-823c-48994c31fd89"). InnerVolumeSpecName "kube-api-access-kdjbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:33:46.161526 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:46.161473    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7719-140153893983552"]
Sep 06 19:33:46.161892 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:46.161878    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19f69a9f-6923-479f-823c-48994c31fd89" containerName="test-curl-7719-140153893983552"
Sep 06 19:33:46.161942 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:46.161893    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="19f69a9f-6923-479f-823c-48994c31fd89" containerName="test-curl-7719-140153893983552"
Sep 06 19:33:46.161978 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:46.161959    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="19f69a9f-6923-479f-823c-48994c31fd89" containerName="test-curl-7719-140153893983552"
Sep 06 19:33:46.164886 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:46.164872    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-140153893983552"
Sep 06 19:33:46.170844 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:46.170820    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7719-140153893983552"]
Sep 06 19:33:46.200803 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:46.200782    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krc5x\" (UniqueName: \"kubernetes.io/projected/5836c2dc-9bee-4628-acf7-a577d6a30164-kube-api-access-krc5x\") pod \"test-curl-7719-140153893983552\" (UID: \"5836c2dc-9bee-4628-acf7-a577d6a30164\") " pod="opendatahub/test-curl-7719-140153893983552"
Sep 06 19:33:46.200900 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:46.200833    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdjbp\" (UniqueName: \"kubernetes.io/projected/19f69a9f-6923-479f-823c-48994c31fd89-kube-api-access-kdjbp\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:33:46.301937 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:46.301905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krc5x\" (UniqueName: \"kubernetes.io/projected/5836c2dc-9bee-4628-acf7-a577d6a30164-kube-api-access-krc5x\") pod \"test-curl-7719-140153893983552\" (UID: \"5836c2dc-9bee-4628-acf7-a577d6a30164\") " pod="opendatahub/test-curl-7719-140153893983552"
Sep 06 19:33:46.309677 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:46.309654    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krc5x\" (UniqueName: \"kubernetes.io/projected/5836c2dc-9bee-4628-acf7-a577d6a30164-kube-api-access-krc5x\") pod \"test-curl-7719-140153893983552\" (UID: \"5836c2dc-9bee-4628-acf7-a577d6a30164\") " pod="opendatahub/test-curl-7719-140153893983552"
Sep 06 19:33:46.311388 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:46.311365    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19f69a9f-6923-479f-823c-48994c31fd89" path="/var/lib/kubelet/pods/19f69a9f-6923-479f-823c-48994c31fd89/volumes"
Sep 06 19:33:46.474963 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:46.474945    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-140153893983552"
Sep 06 19:33:46.746744 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:46.746718    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7719-140153893983552"]
Sep 06 19:33:46.748746 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:33:46.748723    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5836c2dc_9bee_4628_acf7_a577d6a30164.slice/crio-aaafeea0aa4e40f9163e1fdcf06ebb14a06c39e06da88dfc2efd31d638ab2af2 WatchSource:0}: Error finding container aaafeea0aa4e40f9163e1fdcf06ebb14a06c39e06da88dfc2efd31d638ab2af2: Status 404 returned error can't find the container with id aaafeea0aa4e40f9163e1fdcf06ebb14a06c39e06da88dfc2efd31d638ab2af2
Sep 06 19:33:46.874353 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:46.874325    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-140153893983552"
Sep 06 19:33:46.874527 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:46.874366    2566 scope.go:117] "RemoveContainer" containerID="31462b444f7cf237f643079e6c7de89e4703e31b39d03dce52f369226891b4d2"
Sep 06 19:33:46.875848 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:46.875818    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7719-140153893983552" event={"ID":"5836c2dc-9bee-4628-acf7-a577d6a30164","Type":"ContainerStarted","Data":"aaafeea0aa4e40f9163e1fdcf06ebb14a06c39e06da88dfc2efd31d638ab2af2"}
Sep 06 19:33:47.884808 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:47.884769    2566 generic.go:356] "Generic (PLEG): container finished" podID="5836c2dc-9bee-4628-acf7-a577d6a30164" containerID="57006835831fb5221b3380d479bf0bc2aa8903a12290b71fdd763a8c076f2093" exitCode=0
Sep 06 19:33:47.885162 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:47.884833    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7719-140153893983552" event={"ID":"5836c2dc-9bee-4628-acf7-a577d6a30164","Type":"ContainerDied","Data":"57006835831fb5221b3380d479bf0bc2aa8903a12290b71fdd763a8c076f2093"}
Sep 06 19:33:49.025017 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:49.024982    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-140153893983552"
Sep 06 19:33:49.041228 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:49.041206    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7719-140153893983552_5836c2dc-9bee-4628-acf7-a577d6a30164/test-curl-7719-140153893983552/0.log"
Sep 06 19:33:49.062203 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:49.062179    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7719-140153893983552"]
Sep 06 19:33:49.064374 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:49.064355    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7719-140153893983552"]
Sep 06 19:33:49.126159 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:49.126134    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krc5x\" (UniqueName: \"kubernetes.io/projected/5836c2dc-9bee-4628-acf7-a577d6a30164-kube-api-access-krc5x\") pod \"5836c2dc-9bee-4628-acf7-a577d6a30164\" (UID: \"5836c2dc-9bee-4628-acf7-a577d6a30164\") "
Sep 06 19:33:49.128296 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:49.128272    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5836c2dc-9bee-4628-acf7-a577d6a30164-kube-api-access-krc5x" (OuterVolumeSpecName: "kube-api-access-krc5x") pod "5836c2dc-9bee-4628-acf7-a577d6a30164" (UID: "5836c2dc-9bee-4628-acf7-a577d6a30164"). InnerVolumeSpecName "kube-api-access-krc5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:33:49.176246 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:49.176187    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7719-140153893984128"]
Sep 06 19:33:49.176553 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:49.176542    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5836c2dc-9bee-4628-acf7-a577d6a30164" containerName="test-curl-7719-140153893983552"
Sep 06 19:33:49.176592 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:49.176555    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5836c2dc-9bee-4628-acf7-a577d6a30164" containerName="test-curl-7719-140153893983552"
Sep 06 19:33:49.176681 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:49.176669    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5836c2dc-9bee-4628-acf7-a577d6a30164" containerName="test-curl-7719-140153893983552"
Sep 06 19:33:49.179774 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:49.179701    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-140153893984128"
Sep 06 19:33:49.182337 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:49.182316    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7719-140153893984128"]
Sep 06 19:33:49.226739 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:49.226719    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wl5f\" (UniqueName: \"kubernetes.io/projected/8940f135-2f15-4bf5-ba5e-0c849c7f04a0-kube-api-access-8wl5f\") pod \"test-curl-7719-140153893984128\" (UID: \"8940f135-2f15-4bf5-ba5e-0c849c7f04a0\") " pod="opendatahub/test-curl-7719-140153893984128"
Sep 06 19:33:49.226847 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:49.226773    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krc5x\" (UniqueName: \"kubernetes.io/projected/5836c2dc-9bee-4628-acf7-a577d6a30164-kube-api-access-krc5x\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:33:49.328161 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:49.328137    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wl5f\" (UniqueName: \"kubernetes.io/projected/8940f135-2f15-4bf5-ba5e-0c849c7f04a0-kube-api-access-8wl5f\") pod \"test-curl-7719-140153893984128\" (UID: \"8940f135-2f15-4bf5-ba5e-0c849c7f04a0\") " pod="opendatahub/test-curl-7719-140153893984128"
Sep 06 19:33:49.335751 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:49.335724    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wl5f\" (UniqueName: \"kubernetes.io/projected/8940f135-2f15-4bf5-ba5e-0c849c7f04a0-kube-api-access-8wl5f\") pod \"test-curl-7719-140153893984128\" (UID: \"8940f135-2f15-4bf5-ba5e-0c849c7f04a0\") " pod="opendatahub/test-curl-7719-140153893984128"
Sep 06 19:33:49.490085 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:49.490060    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-140153893984128"
Sep 06 19:33:49.756048 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:49.756020    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7719-140153893984128"]
Sep 06 19:33:49.757511 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:33:49.757485    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8940f135_2f15_4bf5_ba5e_0c849c7f04a0.slice/crio-a6661b4f311ac56560ba464ae1e93cbe9c032076b73665ab54072a84d9eaea60 WatchSource:0}: Error finding container a6661b4f311ac56560ba464ae1e93cbe9c032076b73665ab54072a84d9eaea60: Status 404 returned error can't find the container with id a6661b4f311ac56560ba464ae1e93cbe9c032076b73665ab54072a84d9eaea60
Sep 06 19:33:49.897387 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:49.897359    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-140153893983552"
Sep 06 19:33:49.897557 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:49.897410    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaafeea0aa4e40f9163e1fdcf06ebb14a06c39e06da88dfc2efd31d638ab2af2"
Sep 06 19:33:49.898811 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:49.898786    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7719-140153893984128" event={"ID":"8940f135-2f15-4bf5-ba5e-0c849c7f04a0","Type":"ContainerStarted","Data":"a6661b4f311ac56560ba464ae1e93cbe9c032076b73665ab54072a84d9eaea60"}
Sep 06 19:33:49.907901 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:33:49.907880    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7719-140153893983552\" is forbidden: User \"system:node:ip-10-0-129-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-252.ec2.internal' and this object" podUID="5836c2dc-9bee-4628-acf7-a577d6a30164" pod="opendatahub/test-curl-7719-140153893983552"
Sep 06 19:33:50.310482 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:50.310452    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5836c2dc-9bee-4628-acf7-a577d6a30164" path="/var/lib/kubelet/pods/5836c2dc-9bee-4628-acf7-a577d6a30164/volumes"
Sep 06 19:33:50.906212 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:50.906179    2566 generic.go:356] "Generic (PLEG): container finished" podID="8940f135-2f15-4bf5-ba5e-0c849c7f04a0" containerID="517ca8872a7dd15cec39c413c00e88feb32c2ab064952826e401f6e13f847b1c" exitCode=0
Sep 06 19:33:50.906389 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:50.906267    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7719-140153893984128" event={"ID":"8940f135-2f15-4bf5-ba5e-0c849c7f04a0","Type":"ContainerDied","Data":"517ca8872a7dd15cec39c413c00e88feb32c2ab064952826e401f6e13f847b1c"}
Sep 06 19:33:52.038794 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:52.038769    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-140153893984128"
Sep 06 19:33:52.055254 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:52.055214    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7719-140153893984128_8940f135-2f15-4bf5-ba5e-0c849c7f04a0/test-curl-7719-140153893984128/0.log"
Sep 06 19:33:52.076403 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:52.076368    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7719-140153893984128"]
Sep 06 19:33:52.078751 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:52.078734    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7719-140153893984128"]
Sep 06 19:33:52.152253 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:52.152228    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wl5f\" (UniqueName: \"kubernetes.io/projected/8940f135-2f15-4bf5-ba5e-0c849c7f04a0-kube-api-access-8wl5f\") pod \"8940f135-2f15-4bf5-ba5e-0c849c7f04a0\" (UID: \"8940f135-2f15-4bf5-ba5e-0c849c7f04a0\") "
Sep 06 19:33:52.154347 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:52.154317    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8940f135-2f15-4bf5-ba5e-0c849c7f04a0-kube-api-access-8wl5f" (OuterVolumeSpecName: "kube-api-access-8wl5f") pod "8940f135-2f15-4bf5-ba5e-0c849c7f04a0" (UID: "8940f135-2f15-4bf5-ba5e-0c849c7f04a0"). InnerVolumeSpecName "kube-api-access-8wl5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:33:52.192921 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:52.192857    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7719-140153893984128"]
Sep 06 19:33:52.193249 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:52.193236    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8940f135-2f15-4bf5-ba5e-0c849c7f04a0" containerName="test-curl-7719-140153893984128"
Sep 06 19:33:52.193297 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:52.193250    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8940f135-2f15-4bf5-ba5e-0c849c7f04a0" containerName="test-curl-7719-140153893984128"
Sep 06 19:33:52.193333 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:52.193322    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8940f135-2f15-4bf5-ba5e-0c849c7f04a0" containerName="test-curl-7719-140153893984128"
Sep 06 19:33:52.196218 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:52.196204    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-140153893984128"
Sep 06 19:33:52.200193 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:52.200175    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7719-140153893984128"]
Sep 06 19:33:52.253731 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:52.253703    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4jq5\" (UniqueName: \"kubernetes.io/projected/e5113963-9201-4007-a67a-6de4ca095dd0-kube-api-access-l4jq5\") pod \"test-curl-7719-140153893984128\" (UID: \"e5113963-9201-4007-a67a-6de4ca095dd0\") " pod="opendatahub/test-curl-7719-140153893984128"
Sep 06 19:33:52.253833 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:52.253745    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wl5f\" (UniqueName: \"kubernetes.io/projected/8940f135-2f15-4bf5-ba5e-0c849c7f04a0-kube-api-access-8wl5f\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:33:52.310751 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:52.310728    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8940f135-2f15-4bf5-ba5e-0c849c7f04a0" path="/var/lib/kubelet/pods/8940f135-2f15-4bf5-ba5e-0c849c7f04a0/volumes"
Sep 06 19:33:52.354929 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:52.354905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4jq5\" (UniqueName: \"kubernetes.io/projected/e5113963-9201-4007-a67a-6de4ca095dd0-kube-api-access-l4jq5\") pod \"test-curl-7719-140153893984128\" (UID: \"e5113963-9201-4007-a67a-6de4ca095dd0\") " pod="opendatahub/test-curl-7719-140153893984128"
Sep 06 19:33:52.363099 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:52.363072    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4jq5\" (UniqueName: \"kubernetes.io/projected/e5113963-9201-4007-a67a-6de4ca095dd0-kube-api-access-l4jq5\") pod \"test-curl-7719-140153893984128\" (UID: \"e5113963-9201-4007-a67a-6de4ca095dd0\") " pod="opendatahub/test-curl-7719-140153893984128"
Sep 06 19:33:52.506391 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:52.506351    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-140153893984128"
Sep 06 19:33:52.772939 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:52.772914    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7719-140153893984128"]
Sep 06 19:33:52.774347 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:33:52.774321    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5113963_9201_4007_a67a_6de4ca095dd0.slice/crio-0e774cdfcf78af07a52f4ef6afb72c42d507dff9e7a3e51425becb61151f6484 WatchSource:0}: Error finding container 0e774cdfcf78af07a52f4ef6afb72c42d507dff9e7a3e51425becb61151f6484: Status 404 returned error can't find the container with id 0e774cdfcf78af07a52f4ef6afb72c42d507dff9e7a3e51425becb61151f6484
Sep 06 19:33:52.918735 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:52.918706    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-140153893984128"
Sep 06 19:33:52.918735 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:52.918724    2566 scope.go:117] "RemoveContainer" containerID="517ca8872a7dd15cec39c413c00e88feb32c2ab064952826e401f6e13f847b1c"
Sep 06 19:33:52.920077 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:52.920019    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7719-140153893984128" event={"ID":"e5113963-9201-4007-a67a-6de4ca095dd0","Type":"ContainerStarted","Data":"0e774cdfcf78af07a52f4ef6afb72c42d507dff9e7a3e51425becb61151f6484"}
Sep 06 19:33:53.928199 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:53.928167    2566 generic.go:356] "Generic (PLEG): container finished" podID="e5113963-9201-4007-a67a-6de4ca095dd0" containerID="084211757c40446dc132009a6bd6c70f2e5abd855fbdeee25e572aff7a8255fd" exitCode=0
Sep 06 19:33:53.928564 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:53.928219    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7719-140153893984128" event={"ID":"e5113963-9201-4007-a67a-6de4ca095dd0","Type":"ContainerDied","Data":"084211757c40446dc132009a6bd6c70f2e5abd855fbdeee25e572aff7a8255fd"}
Sep 06 19:33:55.066432 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:55.066407    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-140153893984128"
Sep 06 19:33:55.102678 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:55.102658    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7719-140153893984128_e5113963-9201-4007-a67a-6de4ca095dd0/test-curl-7719-140153893984128/0.log"
Sep 06 19:33:55.138358 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:55.138334    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7719-140153893984128"]
Sep 06 19:33:55.144355 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:55.144330    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7719-140153893984128"]
Sep 06 19:33:55.181630 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:55.181595    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4jq5\" (UniqueName: \"kubernetes.io/projected/e5113963-9201-4007-a67a-6de4ca095dd0-kube-api-access-l4jq5\") pod \"e5113963-9201-4007-a67a-6de4ca095dd0\" (UID: \"e5113963-9201-4007-a67a-6de4ca095dd0\") "
Sep 06 19:33:55.183822 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:55.183795    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5113963-9201-4007-a67a-6de4ca095dd0-kube-api-access-l4jq5" (OuterVolumeSpecName: "kube-api-access-l4jq5") pod "e5113963-9201-4007-a67a-6de4ca095dd0" (UID: "e5113963-9201-4007-a67a-6de4ca095dd0"). InnerVolumeSpecName "kube-api-access-l4jq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:33:55.282382 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:55.282345    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4jq5\" (UniqueName: \"kubernetes.io/projected/e5113963-9201-4007-a67a-6de4ca095dd0-kube-api-access-l4jq5\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:33:55.941201 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:55.941169    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-140153893984128"
Sep 06 19:33:55.941201 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:55.941200    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e774cdfcf78af07a52f4ef6afb72c42d507dff9e7a3e51425becb61151f6484"
Sep 06 19:33:55.950697 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:33:55.950672    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7719-140153893984128\" is forbidden: User \"system:node:ip-10-0-129-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-252.ec2.internal' and this object" podUID="e5113963-9201-4007-a67a-6de4ca095dd0" pod="opendatahub/test-curl-7719-140153893984128"
Sep 06 19:33:56.309340 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:33:56.309295    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7719-140153893984128\" is forbidden: User \"system:node:ip-10-0-129-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-252.ec2.internal' and this object" podUID="e5113963-9201-4007-a67a-6de4ca095dd0" pod="opendatahub/test-curl-7719-140153893984128"
Sep 06 19:33:56.310504 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:33:56.310482    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5113963-9201-4007-a67a-6de4ca095dd0" path="/var/lib/kubelet/pods/e5113963-9201-4007-a67a-6de4ca095dd0/volumes"
Sep 06 19:34:00.461134 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:00.461101    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"]
Sep 06 19:34:00.461706 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:00.461690    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5113963-9201-4007-a67a-6de4ca095dd0" containerName="test-curl-7719-140153893984128"
Sep 06 19:34:00.461767 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:00.461709    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5113963-9201-4007-a67a-6de4ca095dd0" containerName="test-curl-7719-140153893984128"
Sep 06 19:34:00.461810 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:00.461798    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5113963-9201-4007-a67a-6de4ca095dd0" containerName="test-curl-7719-140153893984128"
Sep 06 19:34:00.465148 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:00.465129    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"
Sep 06 19:34:00.467715 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:00.467693    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-84ba6496-serving-cert\""
Sep 06 19:34:00.473933 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:00.473908    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"]
Sep 06 19:34:00.529185 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:00.529162    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3c6f539-5d94-4d16-a008-2dacfe6ac568-metrics-certs\") pod \"maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x\" (UID: \"e3c6f539-5d94-4d16-a008-2dacfe6ac568\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"
Sep 06 19:34:00.529311 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:00.529228    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3c6f539-5d94-4d16-a008-2dacfe6ac568-maas-api-tls\") pod \"maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x\" (UID: \"e3c6f539-5d94-4d16-a008-2dacfe6ac568\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"
Sep 06 19:34:00.529311 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:00.529253    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3c6f539-5d94-4d16-a008-2dacfe6ac568-openshift-service-ca\") pod \"maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x\" (UID: \"e3c6f539-5d94-4d16-a008-2dacfe6ac568\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"
Sep 06 19:34:00.529395 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:00.529351    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsdjt\" (UniqueName: \"kubernetes.io/projected/e3c6f539-5d94-4d16-a008-2dacfe6ac568-kube-api-access-qsdjt\") pod \"maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x\" (UID: \"e3c6f539-5d94-4d16-a008-2dacfe6ac568\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"
Sep 06 19:34:00.630208 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:00.630178    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsdjt\" (UniqueName: \"kubernetes.io/projected/e3c6f539-5d94-4d16-a008-2dacfe6ac568-kube-api-access-qsdjt\") pod \"maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x\" (UID: \"e3c6f539-5d94-4d16-a008-2dacfe6ac568\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"
Sep 06 19:34:00.630378 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:00.630226    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3c6f539-5d94-4d16-a008-2dacfe6ac568-metrics-certs\") pod \"maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x\" (UID: \"e3c6f539-5d94-4d16-a008-2dacfe6ac568\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"
Sep 06 19:34:00.630378 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:00.630261    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3c6f539-5d94-4d16-a008-2dacfe6ac568-maas-api-tls\") pod \"maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x\" (UID: \"e3c6f539-5d94-4d16-a008-2dacfe6ac568\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"
Sep 06 19:34:00.630378 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:00.630277    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3c6f539-5d94-4d16-a008-2dacfe6ac568-openshift-service-ca\") pod \"maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x\" (UID: \"e3c6f539-5d94-4d16-a008-2dacfe6ac568\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"
Sep 06 19:34:00.630543 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:34:00.630412    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-serving-cert: secret "maas-api-e2e-mt-84ba6496-serving-cert" not found
Sep 06 19:34:00.630543 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:34:00.630489    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3c6f539-5d94-4d16-a008-2dacfe6ac568-maas-api-tls podName:e3c6f539-5d94-4d16-a008-2dacfe6ac568 nodeName:}" failed. No retries permitted until 2026-09-06 19:34:01.130470509 +0000 UTC m=+1375.311652300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e3c6f539-5d94-4d16-a008-2dacfe6ac568-maas-api-tls") pod "maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x" (UID: "e3c6f539-5d94-4d16-a008-2dacfe6ac568") : secret "maas-api-e2e-mt-84ba6496-serving-cert" not found
Sep 06 19:34:00.630911 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:00.630893    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3c6f539-5d94-4d16-a008-2dacfe6ac568-openshift-service-ca\") pod \"maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x\" (UID: \"e3c6f539-5d94-4d16-a008-2dacfe6ac568\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"
Sep 06 19:34:00.632861 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:00.632841    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3c6f539-5d94-4d16-a008-2dacfe6ac568-metrics-certs\") pod \"maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x\" (UID: \"e3c6f539-5d94-4d16-a008-2dacfe6ac568\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"
Sep 06 19:34:00.641252 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:00.641231    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsdjt\" (UniqueName: \"kubernetes.io/projected/e3c6f539-5d94-4d16-a008-2dacfe6ac568-kube-api-access-qsdjt\") pod \"maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x\" (UID: \"e3c6f539-5d94-4d16-a008-2dacfe6ac568\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"
Sep 06 19:34:01.105504 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.105468    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2"]
Sep 06 19:34:01.110240 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.110217    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2"
Sep 06 19:34:01.112912 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.112888    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-84ba6496\""
Sep 06 19:34:01.113038 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.113003    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-84ba6496-dockercfg-jtj7q\""
Sep 06 19:34:01.114092 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.114075    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2"]
Sep 06 19:34:01.134993 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.134974    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3c6f539-5d94-4d16-a008-2dacfe6ac568-maas-api-tls\") pod \"maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x\" (UID: \"e3c6f539-5d94-4d16-a008-2dacfe6ac568\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"
Sep 06 19:34:01.137436 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.137407    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3c6f539-5d94-4d16-a008-2dacfe6ac568-maas-api-tls\") pod \"maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x\" (UID: \"e3c6f539-5d94-4d16-a008-2dacfe6ac568\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"
Sep 06 19:34:01.236225 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.236197    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bae0b8-b927-4fdc-890e-ff2c1555039b-plugins-config-volume\") pod \"payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2\" (UID: \"35bae0b8-b927-4fdc-890e-ff2c1555039b\") " pod="openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2"
Sep 06 19:34:01.236373 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.236285    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4dpr\" (UniqueName: \"kubernetes.io/projected/35bae0b8-b927-4fdc-890e-ff2c1555039b-kube-api-access-m4dpr\") pod \"payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2\" (UID: \"35bae0b8-b927-4fdc-890e-ff2c1555039b\") " pod="openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2"
Sep 06 19:34:01.337482 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.337452    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bae0b8-b927-4fdc-890e-ff2c1555039b-plugins-config-volume\") pod \"payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2\" (UID: \"35bae0b8-b927-4fdc-890e-ff2c1555039b\") " pod="openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2"
Sep 06 19:34:01.337672 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.337534    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4dpr\" (UniqueName: \"kubernetes.io/projected/35bae0b8-b927-4fdc-890e-ff2c1555039b-kube-api-access-m4dpr\") pod \"payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2\" (UID: \"35bae0b8-b927-4fdc-890e-ff2c1555039b\") " pod="openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2"
Sep 06 19:34:01.338168 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.338147    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bae0b8-b927-4fdc-890e-ff2c1555039b-plugins-config-volume\") pod \"payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2\" (UID: \"35bae0b8-b927-4fdc-890e-ff2c1555039b\") " pod="openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2"
Sep 06 19:34:01.346038 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.346013    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4dpr\" (UniqueName: \"kubernetes.io/projected/35bae0b8-b927-4fdc-890e-ff2c1555039b-kube-api-access-m4dpr\") pod \"payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2\" (UID: \"35bae0b8-b927-4fdc-890e-ff2c1555039b\") " pod="openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2"
Sep 06 19:34:01.351096 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.351072    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"]
Sep 06 19:34:01.354745 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.354725    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"
Sep 06 19:34:01.357079 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.357032    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-476f98-serving-cert\""
Sep 06 19:34:01.363092 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.363072    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"]
Sep 06 19:34:01.379820 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.379794    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"
Sep 06 19:34:01.421469 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.421439    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2"
Sep 06 19:34:01.438596 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.438465    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40fdea58-341d-4f7c-9815-60776ddf99a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h\" (UID: \"40fdea58-341d-4f7c-9815-60776ddf99a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"
Sep 06 19:34:01.438596 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.438515    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcrf5\" (UniqueName: \"kubernetes.io/projected/40fdea58-341d-4f7c-9815-60776ddf99a7-kube-api-access-bcrf5\") pod \"maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h\" (UID: \"40fdea58-341d-4f7c-9815-60776ddf99a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"
Sep 06 19:34:01.438765 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.438750    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40fdea58-341d-4f7c-9815-60776ddf99a7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h\" (UID: \"40fdea58-341d-4f7c-9815-60776ddf99a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"
Sep 06 19:34:01.438823 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.438783    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40fdea58-341d-4f7c-9815-60776ddf99a7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h\" (UID: \"40fdea58-341d-4f7c-9815-60776ddf99a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"
Sep 06 19:34:01.540380 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.540334    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40fdea58-341d-4f7c-9815-60776ddf99a7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h\" (UID: \"40fdea58-341d-4f7c-9815-60776ddf99a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"
Sep 06 19:34:01.540871 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.540390    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40fdea58-341d-4f7c-9815-60776ddf99a7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h\" (UID: \"40fdea58-341d-4f7c-9815-60776ddf99a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"
Sep 06 19:34:01.540871 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.540442    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40fdea58-341d-4f7c-9815-60776ddf99a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h\" (UID: \"40fdea58-341d-4f7c-9815-60776ddf99a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"
Sep 06 19:34:01.540871 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.540486    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcrf5\" (UniqueName: \"kubernetes.io/projected/40fdea58-341d-4f7c-9815-60776ddf99a7-kube-api-access-bcrf5\") pod \"maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h\" (UID: \"40fdea58-341d-4f7c-9815-60776ddf99a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"
Sep 06 19:34:01.541050 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:34:01.540939    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-serving-cert: secret "maas-api-e2e-shared-b-w3-476f98-serving-cert" not found
Sep 06 19:34:01.541050 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:34:01.541011    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40fdea58-341d-4f7c-9815-60776ddf99a7-maas-api-tls podName:40fdea58-341d-4f7c-9815-60776ddf99a7 nodeName:}" failed. No retries permitted until 2026-09-06 19:34:02.04099101 +0000 UTC m=+1376.222172800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/40fdea58-341d-4f7c-9815-60776ddf99a7-maas-api-tls") pod "maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h" (UID: "40fdea58-341d-4f7c-9815-60776ddf99a7") : secret "maas-api-e2e-shared-b-w3-476f98-serving-cert" not found
Sep 06 19:34:01.541178 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.541156    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40fdea58-341d-4f7c-9815-60776ddf99a7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h\" (UID: \"40fdea58-341d-4f7c-9815-60776ddf99a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"
Sep 06 19:34:01.543018 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.542991    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40fdea58-341d-4f7c-9815-60776ddf99a7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h\" (UID: \"40fdea58-341d-4f7c-9815-60776ddf99a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"
Sep 06 19:34:01.548873 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.548848    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcrf5\" (UniqueName: \"kubernetes.io/projected/40fdea58-341d-4f7c-9815-60776ddf99a7-kube-api-access-bcrf5\") pod \"maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h\" (UID: \"40fdea58-341d-4f7c-9815-60776ddf99a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"
Sep 06 19:34:01.663011 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.662987    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"]
Sep 06 19:34:01.664838 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:34:01.664813    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c6f539_5d94_4d16_a008_2dacfe6ac568.slice/crio-58b64851f44e12d461f87d1490aa3ab09700c08cba60af97669be1d1c7408827 WatchSource:0}: Error finding container 58b64851f44e12d461f87d1490aa3ab09700c08cba60af97669be1d1c7408827: Status 404 returned error can't find the container with id 58b64851f44e12d461f87d1490aa3ab09700c08cba60af97669be1d1c7408827
Sep 06 19:34:01.714598 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.714577    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2"]
Sep 06 19:34:01.716479 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:34:01.716457    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35bae0b8_b927_4fdc_890e_ff2c1555039b.slice/crio-d8d9a0fbbc009aa2ada2f00abf3b238b81ed9a34458030685490de2519849d8b WatchSource:0}: Error finding container d8d9a0fbbc009aa2ada2f00abf3b238b81ed9a34458030685490de2519849d8b: Status 404 returned error can't find the container with id d8d9a0fbbc009aa2ada2f00abf3b238b81ed9a34458030685490de2519849d8b
Sep 06 19:34:01.975507 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.975471    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x" event={"ID":"e3c6f539-5d94-4d16-a008-2dacfe6ac568","Type":"ContainerStarted","Data":"58b64851f44e12d461f87d1490aa3ab09700c08cba60af97669be1d1c7408827"}
Sep 06 19:34:01.977173 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.977149    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2" event={"ID":"35bae0b8-b927-4fdc-890e-ff2c1555039b","Type":"ContainerStarted","Data":"76f2411bde38995847a2aa66d0e0c2ba068dafab3f2fe6f83f96d5431f581605"}
Sep 06 19:34:01.977304 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.977192    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2" event={"ID":"35bae0b8-b927-4fdc-890e-ff2c1555039b","Type":"ContainerStarted","Data":"d8d9a0fbbc009aa2ada2f00abf3b238b81ed9a34458030685490de2519849d8b"}
Sep 06 19:34:01.977304 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.977236    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2"
Sep 06 19:34:01.994111 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:01.994062    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2" podStartSLOduration=0.994048058 podStartE2EDuration="994.048058ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:34:01.992451413 +0000 UTC m=+1376.173633219" watchObservedRunningTime="2026-09-06 19:34:01.994048058 +0000 UTC m=+1376.175229911"
Sep 06 19:34:02.045374 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:02.045349    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40fdea58-341d-4f7c-9815-60776ddf99a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h\" (UID: \"40fdea58-341d-4f7c-9815-60776ddf99a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"
Sep 06 19:34:02.048098 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:02.048077    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40fdea58-341d-4f7c-9815-60776ddf99a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h\" (UID: \"40fdea58-341d-4f7c-9815-60776ddf99a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"
Sep 06 19:34:02.266072 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:02.266000    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"
Sep 06 19:34:02.547528 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:02.547506    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"]
Sep 06 19:34:02.548886 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:34:02.548862    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40fdea58_341d_4f7c_9815_60776ddf99a7.slice/crio-d8d1b00152d260c7e5262e37afd13a0e6d1e8444f1d9f83bf5f3368fbec8ef36 WatchSource:0}: Error finding container d8d1b00152d260c7e5262e37afd13a0e6d1e8444f1d9f83bf5f3368fbec8ef36: Status 404 returned error can't find the container with id d8d1b00152d260c7e5262e37afd13a0e6d1e8444f1d9f83bf5f3368fbec8ef36
Sep 06 19:34:02.985650 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:02.985594    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x" event={"ID":"e3c6f539-5d94-4d16-a008-2dacfe6ac568","Type":"ContainerStarted","Data":"cb089a0e9d2c194ab0f6a2d95f9989f4bc07c672d5f8c37c791c16329b7ae70f"}
Sep 06 19:34:02.985814 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:02.985707    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"
Sep 06 19:34:02.987314 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:02.987275    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h" event={"ID":"40fdea58-341d-4f7c-9815-60776ddf99a7","Type":"ContainerStarted","Data":"d8d1b00152d260c7e5262e37afd13a0e6d1e8444f1d9f83bf5f3368fbec8ef36"}
Sep 06 19:34:03.003052 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:03.003011    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x" podStartSLOduration=2.6262650450000002 podStartE2EDuration="3.002997964s" totalImagesPullingTime="376.732919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:34:03.000794736 +0000 UTC m=+1377.181976543" watchObservedRunningTime="2026-09-06 19:34:03.002997964 +0000 UTC m=+1377.184179770"
Sep 06 19:34:03.995173 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:03.995134    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h" event={"ID":"40fdea58-341d-4f7c-9815-60776ddf99a7","Type":"ContainerStarted","Data":"a18a8b7ee5b51e020febee0db8a095e1312904dfd8e62518f213ecbe96f11969"}
Sep 06 19:34:03.995584 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:03.995202    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"
Sep 06 19:34:04.012835 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:04.012792    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h" podStartSLOduration=2.537042782 podStartE2EDuration="3.012766875s" totalImagesPullingTime="475.724093ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:34:04.010285369 +0000 UTC m=+1378.191467175" watchObservedRunningTime="2026-09-06 19:34:04.012766875 +0000 UTC m=+1378.193948683"
Sep 06 19:34:06.473706 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:06.473677    2566 scope.go:117] "RemoveContainer" containerID="42aea4649b564c95563d528fceda93f2e3828f883327ee24e2e829dec7613cdf"
Sep 06 19:34:08.064919 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:08.064877    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:34:09.001272 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:09.001239    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"
Sep 06 19:34:10.007398 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:10.007372    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"
Sep 06 19:34:12.989275 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:12.989240    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2"
Sep 06 19:34:31.042800 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.042764    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"]
Sep 06 19:34:31.048255 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.048237    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:31.051927 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.051906    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-84ba6496\"/\"kube-root-ca.crt\""
Sep 06 19:34:31.052063 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.051986    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-84ba6496\"/\"default-dockercfg-b6rfl\""
Sep 06 19:34:31.052063 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.052035    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-84ba6496\"/\"openshift-service-ca.crt\""
Sep 06 19:34:31.052063 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.051985    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-84ba6496\"/\"e2e-lifecycle-model-84ba6496-kserve-self-signed-certs\""
Sep 06 19:34:31.054873 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.054849    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"]
Sep 06 19:34:31.114253 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.114228    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-model-cache\") pod \"e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") " pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:31.114359 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.114259    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-tmp-dir\") pod \"e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") " pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:31.114359 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.114282    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njsvc\" (UniqueName: \"kubernetes.io/projected/c3558bcb-30b1-4990-9ed8-596cecfc9139-kube-api-access-njsvc\") pod \"e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") " pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:31.114432 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.114379    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3558bcb-30b1-4990-9ed8-596cecfc9139-tls-certs\") pod \"e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") " pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:31.114432 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.114414    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-dshm\") pod \"e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") " pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:31.114502 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.114448    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-home\") pod \"e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") " pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:31.215239 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.215211    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-model-cache\") pod \"e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") " pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:31.215357 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.215243    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-tmp-dir\") pod \"e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") " pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:31.215357 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.215266    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njsvc\" (UniqueName: \"kubernetes.io/projected/c3558bcb-30b1-4990-9ed8-596cecfc9139-kube-api-access-njsvc\") pod \"e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") " pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:31.215357 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.215308    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3558bcb-30b1-4990-9ed8-596cecfc9139-tls-certs\") pod \"e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") " pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:31.215357 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.215331    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-dshm\") pod \"e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") " pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:31.215357 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.215353    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-home\") pod \"e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") " pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:31.215682 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.215656    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-model-cache\") pod \"e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") " pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:31.215747 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.215685    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-tmp-dir\") pod \"e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") " pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:31.215747 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.215711    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-home\") pod \"e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") " pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:31.217704 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.217684    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-dshm\") pod \"e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") " pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:31.218028 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.218011    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3558bcb-30b1-4990-9ed8-596cecfc9139-tls-certs\") pod \"e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") " pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:31.222630 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.222595    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njsvc\" (UniqueName: \"kubernetes.io/projected/c3558bcb-30b1-4990-9ed8-596cecfc9139-kube-api-access-njsvc\") pod \"e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") " pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:31.359124 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.359040    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:31.632970 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:31.632945    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"]
Sep 06 19:34:31.635064 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:34:31.635037    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3558bcb_30b1_4990_9ed8_596cecfc9139.slice/crio-7822025b957bcece6027875633f0919721cfe1118201773d59183627a47a9ee6 WatchSource:0}: Error finding container 7822025b957bcece6027875633f0919721cfe1118201773d59183627a47a9ee6: Status 404 returned error can't find the container with id 7822025b957bcece6027875633f0919721cfe1118201773d59183627a47a9ee6
Sep 06 19:34:32.171320 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:32.171284    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk" event={"ID":"c3558bcb-30b1-4990-9ed8-596cecfc9139","Type":"ContainerStarted","Data":"8563a96adca08c59f6027a72dc2f162e8d9fb1174efffc567cad462bda451a8e"}
Sep 06 19:34:32.171320 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:32.171328    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk" event={"ID":"c3558bcb-30b1-4990-9ed8-596cecfc9139","Type":"ContainerStarted","Data":"7822025b957bcece6027875633f0919721cfe1118201773d59183627a47a9ee6"}
Sep 06 19:34:32.187169 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:32.187127    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk" podStartSLOduration=1.187112371 podStartE2EDuration="1.187112371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:34:32.185475626 +0000 UTC m=+1406.366657432" watchObservedRunningTime="2026-09-06 19:34:32.187112371 +0000 UTC m=+1406.368294208"
Sep 06 19:34:36.058153 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:36.058120    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:34:41.359409 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:41.359370    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:41.359409 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:41.359417    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:41.372623 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:41.372589    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:42.240954 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:42.240922    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:34:47.842160 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:47.842126    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"]
Sep 06 19:34:47.845920 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:47.845902    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:34:47.850275 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:47.850248    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e5ef25\"/\"default-dockercfg-ppmt5\""
Sep 06 19:34:47.850671 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:47.850645    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e5ef25\"/\"kube-root-ca.crt\""
Sep 06 19:34:47.853507 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:47.850793    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e5ef25\"/\"auth-test-model-e5ef25-kserve-self-signed-certs\""
Sep 06 19:34:47.853507 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:47.851252    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e5ef25\"/\"openshift-service-ca.crt\""
Sep 06 19:34:47.859503 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:47.859482    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"]
Sep 06 19:34:47.961177 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:47.961153    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-model-cache\") pod \"auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:34:47.961297 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:47.961189    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b118acb5-c68d-4a81-8698-44c4e81286d1-tls-certs\") pod \"auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:34:47.961297 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:47.961217    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-tmp-dir\") pod \"auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:34:47.961373 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:47.961290    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f7qf\" (UniqueName: \"kubernetes.io/projected/b118acb5-c68d-4a81-8698-44c4e81286d1-kube-api-access-8f7qf\") pod \"auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:34:47.961373 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:47.961332    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-dshm\") pod \"auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:34:47.961373 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:47.961349    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-home\") pod \"auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:34:48.061951 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:48.061917    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-home\") pod \"auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:34:48.062087 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:48.062001    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-model-cache\") pod \"auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:34:48.062087 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:48.062039    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b118acb5-c68d-4a81-8698-44c4e81286d1-tls-certs\") pod \"auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:34:48.062198 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:48.062083    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-tmp-dir\") pod \"auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:34:48.062198 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:48.062129    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f7qf\" (UniqueName: \"kubernetes.io/projected/b118acb5-c68d-4a81-8698-44c4e81286d1-kube-api-access-8f7qf\") pod \"auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:34:48.062198 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:48.062171    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-dshm\") pod \"auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:34:48.062415 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:48.062390    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-home\") pod \"auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:34:48.062415 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:48.062410    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-model-cache\") pod \"auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:34:48.062542 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:48.062460    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-tmp-dir\") pod \"auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:34:48.064546 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:48.064521    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-dshm\") pod \"auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:34:48.064743 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:48.064727    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b118acb5-c68d-4a81-8698-44c4e81286d1-tls-certs\") pod \"auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:34:48.069540 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:48.069523    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f7qf\" (UniqueName: \"kubernetes.io/projected/b118acb5-c68d-4a81-8698-44c4e81286d1-kube-api-access-8f7qf\") pod \"auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:34:48.159504 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:48.159448    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:34:48.432383 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:48.432359    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"]
Sep 06 19:34:48.433927 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:34:48.433883    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb118acb5_c68d_4a81_8698_44c4e81286d1.slice/crio-ac815859a728863d419a75811e2c2deb34a79f8fe68c39909cc095c75d3097bc WatchSource:0}: Error finding container ac815859a728863d419a75811e2c2deb34a79f8fe68c39909cc095c75d3097bc: Status 404 returned error can't find the container with id ac815859a728863d419a75811e2c2deb34a79f8fe68c39909cc095c75d3097bc
Sep 06 19:34:49.271100 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:49.271061    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn" event={"ID":"b118acb5-c68d-4a81-8698-44c4e81286d1","Type":"ContainerStarted","Data":"06551203e4d7cda5cf66b224e2bb865233f67ab0c267a50a19610f6d8c2f0ce6"}
Sep 06 19:34:49.271462 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:49.271111    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn" event={"ID":"b118acb5-c68d-4a81-8698-44c4e81286d1","Type":"ContainerStarted","Data":"ac815859a728863d419a75811e2c2deb34a79f8fe68c39909cc095c75d3097bc"}
Sep 06 19:34:49.287291 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:49.287246    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn" podStartSLOduration=2.287231263 podStartE2EDuration="2.287231263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:34:49.285914076 +0000 UTC m=+1423.467095883" watchObservedRunningTime="2026-09-06 19:34:49.287231263 +0000 UTC m=+1423.468413067"
Sep 06 19:34:52.559115 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:52.559080    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:34:58.160061 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:58.159967    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:34:58.160441 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:58.160060    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:34:58.172679 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:58.172658    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:34:58.335394 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:34:58.335364    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:35:04.557063 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:04.557034    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:35:24.039672 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.039641    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"]
Sep 06 19:35:24.040237 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.039860    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x" podUID="e3c6f539-5d94-4d16-a008-2dacfe6ac568" containerName="maas-api" containerID="cri-o://cb089a0e9d2c194ab0f6a2d95f9989f4bc07c672d5f8c37c791c16329b7ae70f" gracePeriod=30
Sep 06 19:35:24.090492 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.090459    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2"]
Sep 06 19:35:24.090760 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.090729    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2" podUID="35bae0b8-b927-4fdc-890e-ff2c1555039b" containerName="payload-processing" containerID="cri-o://76f2411bde38995847a2aa66d0e0c2ba068dafab3f2fe6f83f96d5431f581605" gracePeriod=30
Sep 06 19:35:24.338653 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.338629    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"
Sep 06 19:35:24.367927 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.367906    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2"
Sep 06 19:35:24.382206 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.382185    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bae0b8-b927-4fdc-890e-ff2c1555039b-plugins-config-volume\") pod \"35bae0b8-b927-4fdc-890e-ff2c1555039b\" (UID: \"35bae0b8-b927-4fdc-890e-ff2c1555039b\") "
Sep 06 19:35:24.382323 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.382241    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4dpr\" (UniqueName: \"kubernetes.io/projected/35bae0b8-b927-4fdc-890e-ff2c1555039b-kube-api-access-m4dpr\") pod \"35bae0b8-b927-4fdc-890e-ff2c1555039b\" (UID: \"35bae0b8-b927-4fdc-890e-ff2c1555039b\") "
Sep 06 19:35:24.382323 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.382302    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3c6f539-5d94-4d16-a008-2dacfe6ac568-openshift-service-ca\") pod \"e3c6f539-5d94-4d16-a008-2dacfe6ac568\" (UID: \"e3c6f539-5d94-4d16-a008-2dacfe6ac568\") "
Sep 06 19:35:24.382439 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.382348    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3c6f539-5d94-4d16-a008-2dacfe6ac568-maas-api-tls\") pod \"e3c6f539-5d94-4d16-a008-2dacfe6ac568\" (UID: \"e3c6f539-5d94-4d16-a008-2dacfe6ac568\") "
Sep 06 19:35:24.382439 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.382376    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsdjt\" (UniqueName: \"kubernetes.io/projected/e3c6f539-5d94-4d16-a008-2dacfe6ac568-kube-api-access-qsdjt\") pod \"e3c6f539-5d94-4d16-a008-2dacfe6ac568\" (UID: \"e3c6f539-5d94-4d16-a008-2dacfe6ac568\") "
Sep 06 19:35:24.382549 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.382452    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3c6f539-5d94-4d16-a008-2dacfe6ac568-metrics-certs\") pod \"e3c6f539-5d94-4d16-a008-2dacfe6ac568\" (UID: \"e3c6f539-5d94-4d16-a008-2dacfe6ac568\") "
Sep 06 19:35:24.382690 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.382654    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35bae0b8-b927-4fdc-890e-ff2c1555039b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "35bae0b8-b927-4fdc-890e-ff2c1555039b" (UID: "35bae0b8-b927-4fdc-890e-ff2c1555039b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:35:24.382690 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.382678    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c6f539-5d94-4d16-a008-2dacfe6ac568-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e3c6f539-5d94-4d16-a008-2dacfe6ac568" (UID: "e3c6f539-5d94-4d16-a008-2dacfe6ac568"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:35:24.382888 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.382800    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3c6f539-5d94-4d16-a008-2dacfe6ac568-openshift-service-ca\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:35:24.382888 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.382821    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bae0b8-b927-4fdc-890e-ff2c1555039b-plugins-config-volume\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:35:24.385295 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.385268    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3c6f539-5d94-4d16-a008-2dacfe6ac568-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e3c6f539-5d94-4d16-a008-2dacfe6ac568" (UID: "e3c6f539-5d94-4d16-a008-2dacfe6ac568"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:35:24.385425 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.385369    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3c6f539-5d94-4d16-a008-2dacfe6ac568-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e3c6f539-5d94-4d16-a008-2dacfe6ac568" (UID: "e3c6f539-5d94-4d16-a008-2dacfe6ac568"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:35:24.385596 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.385576    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35bae0b8-b927-4fdc-890e-ff2c1555039b-kube-api-access-m4dpr" (OuterVolumeSpecName: "kube-api-access-m4dpr") pod "35bae0b8-b927-4fdc-890e-ff2c1555039b" (UID: "35bae0b8-b927-4fdc-890e-ff2c1555039b"). InnerVolumeSpecName "kube-api-access-m4dpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:35:24.386092 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.386071    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c6f539-5d94-4d16-a008-2dacfe6ac568-kube-api-access-qsdjt" (OuterVolumeSpecName: "kube-api-access-qsdjt") pod "e3c6f539-5d94-4d16-a008-2dacfe6ac568" (UID: "e3c6f539-5d94-4d16-a008-2dacfe6ac568"). InnerVolumeSpecName "kube-api-access-qsdjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:35:24.480306 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.480256    2566 generic.go:356] "Generic (PLEG): container finished" podID="e3c6f539-5d94-4d16-a008-2dacfe6ac568" containerID="cb089a0e9d2c194ab0f6a2d95f9989f4bc07c672d5f8c37c791c16329b7ae70f" exitCode=0
Sep 06 19:35:24.480434 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.480395    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"
Sep 06 19:35:24.480434 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.480409    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x" event={"ID":"e3c6f539-5d94-4d16-a008-2dacfe6ac568","Type":"ContainerDied","Data":"cb089a0e9d2c194ab0f6a2d95f9989f4bc07c672d5f8c37c791c16329b7ae70f"}
Sep 06 19:35:24.480539 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.480515    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x" event={"ID":"e3c6f539-5d94-4d16-a008-2dacfe6ac568","Type":"ContainerDied","Data":"58b64851f44e12d461f87d1490aa3ab09700c08cba60af97669be1d1c7408827"}
Sep 06 19:35:24.480595 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.480567    2566 scope.go:117] "RemoveContainer" containerID="cb089a0e9d2c194ab0f6a2d95f9989f4bc07c672d5f8c37c791c16329b7ae70f"
Sep 06 19:35:24.482078 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.482054    2566 generic.go:356] "Generic (PLEG): container finished" podID="35bae0b8-b927-4fdc-890e-ff2c1555039b" containerID="76f2411bde38995847a2aa66d0e0c2ba068dafab3f2fe6f83f96d5431f581605" exitCode=0
Sep 06 19:35:24.482200 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.482107    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2"
Sep 06 19:35:24.482200 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.482121    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2" event={"ID":"35bae0b8-b927-4fdc-890e-ff2c1555039b","Type":"ContainerDied","Data":"76f2411bde38995847a2aa66d0e0c2ba068dafab3f2fe6f83f96d5431f581605"}
Sep 06 19:35:24.482200 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.482152    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2" event={"ID":"35bae0b8-b927-4fdc-890e-ff2c1555039b","Type":"ContainerDied","Data":"d8d9a0fbbc009aa2ada2f00abf3b238b81ed9a34458030685490de2519849d8b"}
Sep 06 19:35:24.483301 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.483273    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4dpr\" (UniqueName: \"kubernetes.io/projected/35bae0b8-b927-4fdc-890e-ff2c1555039b-kube-api-access-m4dpr\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:35:24.483301 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.483301    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3c6f539-5d94-4d16-a008-2dacfe6ac568-maas-api-tls\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:35:24.483475 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.483316    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsdjt\" (UniqueName: \"kubernetes.io/projected/e3c6f539-5d94-4d16-a008-2dacfe6ac568-kube-api-access-qsdjt\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:35:24.483475 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.483329    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3c6f539-5d94-4d16-a008-2dacfe6ac568-metrics-certs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:35:24.491816 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.491799    2566 scope.go:117] "RemoveContainer" containerID="cb089a0e9d2c194ab0f6a2d95f9989f4bc07c672d5f8c37c791c16329b7ae70f"
Sep 06 19:35:24.492088 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:35:24.492070    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb089a0e9d2c194ab0f6a2d95f9989f4bc07c672d5f8c37c791c16329b7ae70f\": container with ID starting with cb089a0e9d2c194ab0f6a2d95f9989f4bc07c672d5f8c37c791c16329b7ae70f not found: ID does not exist" containerID="cb089a0e9d2c194ab0f6a2d95f9989f4bc07c672d5f8c37c791c16329b7ae70f"
Sep 06 19:35:24.492156 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.492096    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb089a0e9d2c194ab0f6a2d95f9989f4bc07c672d5f8c37c791c16329b7ae70f"} err="failed to get container status \"cb089a0e9d2c194ab0f6a2d95f9989f4bc07c672d5f8c37c791c16329b7ae70f\": rpc error: code = NotFound desc = could not find container \"cb089a0e9d2c194ab0f6a2d95f9989f4bc07c672d5f8c37c791c16329b7ae70f\": container with ID starting with cb089a0e9d2c194ab0f6a2d95f9989f4bc07c672d5f8c37c791c16329b7ae70f not found: ID does not exist"
Sep 06 19:35:24.492156 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.492113    2566 scope.go:117] "RemoveContainer" containerID="76f2411bde38995847a2aa66d0e0c2ba068dafab3f2fe6f83f96d5431f581605"
Sep 06 19:35:24.500350 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.500333    2566 scope.go:117] "RemoveContainer" containerID="76f2411bde38995847a2aa66d0e0c2ba068dafab3f2fe6f83f96d5431f581605"
Sep 06 19:35:24.500591 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:35:24.500575    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76f2411bde38995847a2aa66d0e0c2ba068dafab3f2fe6f83f96d5431f581605\": container with ID starting with 76f2411bde38995847a2aa66d0e0c2ba068dafab3f2fe6f83f96d5431f581605 not found: ID does not exist" containerID="76f2411bde38995847a2aa66d0e0c2ba068dafab3f2fe6f83f96d5431f581605"
Sep 06 19:35:24.500676 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.500596    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76f2411bde38995847a2aa66d0e0c2ba068dafab3f2fe6f83f96d5431f581605"} err="failed to get container status \"76f2411bde38995847a2aa66d0e0c2ba068dafab3f2fe6f83f96d5431f581605\": rpc error: code = NotFound desc = could not find container \"76f2411bde38995847a2aa66d0e0c2ba068dafab3f2fe6f83f96d5431f581605\": container with ID starting with 76f2411bde38995847a2aa66d0e0c2ba068dafab3f2fe6f83f96d5431f581605 not found: ID does not exist"
Sep 06 19:35:24.504497 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.504478    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"]
Sep 06 19:35:24.508377 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.508352    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84ba6496-776c5dd4bf-2r84x"]
Sep 06 19:35:24.520002 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.519980    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2"]
Sep 06 19:35:24.526061 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.526032    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84ba6496-7bc4f64b9c-dfpv2"]
Sep 06 19:35:24.844618 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:24.844571    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:35:26.311970 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:26.311939    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35bae0b8-b927-4fdc-890e-ff2c1555039b" path="/var/lib/kubelet/pods/35bae0b8-b927-4fdc-890e-ff2c1555039b/volumes"
Sep 06 19:35:26.312317 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:26.312308    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3c6f539-5d94-4d16-a008-2dacfe6ac568" path="/var/lib/kubelet/pods/e3c6f539-5d94-4d16-a008-2dacfe6ac568/volumes"
Sep 06 19:35:32.223514 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:32.223477    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"]
Sep 06 19:35:37.851196 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:37.851157    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:35:47.244339 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.244276    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk" podUID="c3558bcb-30b1-4990-9ed8-596cecfc9139" containerName="main" containerID="cri-o://8563a96adca08c59f6027a72dc2f162e8d9fb1174efffc567cad462bda451a8e" gracePeriod=45
Sep 06 19:35:47.487559 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.487535    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:35:47.597851 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.597779    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-tmp-dir\") pod \"c3558bcb-30b1-4990-9ed8-596cecfc9139\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") "
Sep 06 19:35:47.597978 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.597860    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-dshm\") pod \"c3558bcb-30b1-4990-9ed8-596cecfc9139\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") "
Sep 06 19:35:47.597978 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.597894    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-model-cache\") pod \"c3558bcb-30b1-4990-9ed8-596cecfc9139\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") "
Sep 06 19:35:47.597978 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.597933    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-home\") pod \"c3558bcb-30b1-4990-9ed8-596cecfc9139\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") "
Sep 06 19:35:47.597978 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.597960    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3558bcb-30b1-4990-9ed8-596cecfc9139-tls-certs\") pod \"c3558bcb-30b1-4990-9ed8-596cecfc9139\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") "
Sep 06 19:35:47.598160 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.598007    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njsvc\" (UniqueName: \"kubernetes.io/projected/c3558bcb-30b1-4990-9ed8-596cecfc9139-kube-api-access-njsvc\") pod \"c3558bcb-30b1-4990-9ed8-596cecfc9139\" (UID: \"c3558bcb-30b1-4990-9ed8-596cecfc9139\") "
Sep 06 19:35:47.598160 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.598057    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c3558bcb-30b1-4990-9ed8-596cecfc9139" (UID: "c3558bcb-30b1-4990-9ed8-596cecfc9139"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:35:47.598272 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.598142    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-model-cache" (OuterVolumeSpecName: "model-cache") pod "c3558bcb-30b1-4990-9ed8-596cecfc9139" (UID: "c3558bcb-30b1-4990-9ed8-596cecfc9139"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:35:47.598272 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.598214    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-home" (OuterVolumeSpecName: "home") pod "c3558bcb-30b1-4990-9ed8-596cecfc9139" (UID: "c3558bcb-30b1-4990-9ed8-596cecfc9139"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:35:47.598378 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.598363    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-home\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:35:47.598444 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.598380    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-tmp-dir\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:35:47.598444 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.598390    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-model-cache\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:35:47.600225 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.600203    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-dshm" (OuterVolumeSpecName: "dshm") pod "c3558bcb-30b1-4990-9ed8-596cecfc9139" (UID: "c3558bcb-30b1-4990-9ed8-596cecfc9139"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:35:47.600347 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.600269    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3558bcb-30b1-4990-9ed8-596cecfc9139-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c3558bcb-30b1-4990-9ed8-596cecfc9139" (UID: "c3558bcb-30b1-4990-9ed8-596cecfc9139"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:35:47.600347 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.600279    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3558bcb-30b1-4990-9ed8-596cecfc9139-kube-api-access-njsvc" (OuterVolumeSpecName: "kube-api-access-njsvc") pod "c3558bcb-30b1-4990-9ed8-596cecfc9139" (UID: "c3558bcb-30b1-4990-9ed8-596cecfc9139"). InnerVolumeSpecName "kube-api-access-njsvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:35:47.623289 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.623265    2566 generic.go:356] "Generic (PLEG): container finished" podID="c3558bcb-30b1-4990-9ed8-596cecfc9139" containerID="8563a96adca08c59f6027a72dc2f162e8d9fb1174efffc567cad462bda451a8e" exitCode=0
Sep 06 19:35:47.623388 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.623299    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"
Sep 06 19:35:47.623388 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.623340    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk" event={"ID":"c3558bcb-30b1-4990-9ed8-596cecfc9139","Type":"ContainerDied","Data":"8563a96adca08c59f6027a72dc2f162e8d9fb1174efffc567cad462bda451a8e"}
Sep 06 19:35:47.623388 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.623380    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk" event={"ID":"c3558bcb-30b1-4990-9ed8-596cecfc9139","Type":"ContainerDied","Data":"7822025b957bcece6027875633f0919721cfe1118201773d59183627a47a9ee6"}
Sep 06 19:35:47.623485 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.623400    2566 scope.go:117] "RemoveContainer" containerID="8563a96adca08c59f6027a72dc2f162e8d9fb1174efffc567cad462bda451a8e"
Sep 06 19:35:47.632671 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.632653    2566 scope.go:117] "RemoveContainer" containerID="8563a96adca08c59f6027a72dc2f162e8d9fb1174efffc567cad462bda451a8e"
Sep 06 19:35:47.632945 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:35:47.632923    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8563a96adca08c59f6027a72dc2f162e8d9fb1174efffc567cad462bda451a8e\": container with ID starting with 8563a96adca08c59f6027a72dc2f162e8d9fb1174efffc567cad462bda451a8e not found: ID does not exist" containerID="8563a96adca08c59f6027a72dc2f162e8d9fb1174efffc567cad462bda451a8e"
Sep 06 19:35:47.632988 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.632955    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8563a96adca08c59f6027a72dc2f162e8d9fb1174efffc567cad462bda451a8e"} err="failed to get container status \"8563a96adca08c59f6027a72dc2f162e8d9fb1174efffc567cad462bda451a8e\": rpc error: code = NotFound desc = could not find container \"8563a96adca08c59f6027a72dc2f162e8d9fb1174efffc567cad462bda451a8e\": container with ID starting with 8563a96adca08c59f6027a72dc2f162e8d9fb1174efffc567cad462bda451a8e not found: ID does not exist"
Sep 06 19:35:47.643551 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.643531    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"]
Sep 06 19:35:47.645785 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.645767    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-84ba6496/e2e-lifecycle-model-84ba6496-kserve-687b9769bc-85kdk"]
Sep 06 19:35:47.699139 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.699117    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3558bcb-30b1-4990-9ed8-596cecfc9139-tls-certs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:35:47.699139 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.699137    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njsvc\" (UniqueName: \"kubernetes.io/projected/c3558bcb-30b1-4990-9ed8-596cecfc9139-kube-api-access-njsvc\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:35:47.699264 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:47.699147    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3558bcb-30b1-4990-9ed8-596cecfc9139-dshm\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:35:48.316108 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:48.316075    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3558bcb-30b1-4990-9ed8-596cecfc9139" path="/var/lib/kubelet/pods/c3558bcb-30b1-4990-9ed8-596cecfc9139/volumes"
Sep 06 19:35:49.446315 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:35:49.446282    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:36:04.645278 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:36:04.645236    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:36:06.424544 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:36:06.424514    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnssq_9254e6f1-b73d-49cc-8bc1-513925ccde4f/ovn-acl-logging/0.log"
Sep 06 19:36:06.429456 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:36:06.429438    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnssq_9254e6f1-b73d-49cc-8bc1-513925ccde4f/ovn-acl-logging/0.log"
Sep 06 19:36:06.546333 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:36:06.546308    2566 scope.go:117] "RemoveContainer" containerID="5cdcea514930b52f6fdeaa261834b520f4a348ae81922badda21afedbef2930a"
Sep 06 19:36:06.554983 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:36:06.554968    2566 scope.go:117] "RemoveContainer" containerID="1fe557b7016e654ec7dbde1455783958cb8113f62bc88266860ffe6db74a1b6b"
Sep 06 19:36:22.748113 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:36:22.748079    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:36:38.341998 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:36:38.341927    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:36:54.546238 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:36:54.546204    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:37:11.658546 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:11.658510    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:37:12.768252 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.768216    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"]
Sep 06 19:37:12.768679 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.768664    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3c6f539-5d94-4d16-a008-2dacfe6ac568" containerName="maas-api"
Sep 06 19:37:12.768733 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.768681    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3c6f539-5d94-4d16-a008-2dacfe6ac568" containerName="maas-api"
Sep 06 19:37:12.768733 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.768700    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35bae0b8-b927-4fdc-890e-ff2c1555039b" containerName="payload-processing"
Sep 06 19:37:12.768733 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.768705    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="35bae0b8-b927-4fdc-890e-ff2c1555039b" containerName="payload-processing"
Sep 06 19:37:12.768733 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.768718    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3558bcb-30b1-4990-9ed8-596cecfc9139" containerName="main"
Sep 06 19:37:12.768733 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.768724    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3558bcb-30b1-4990-9ed8-596cecfc9139" containerName="main"
Sep 06 19:37:12.768887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.768800    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="35bae0b8-b927-4fdc-890e-ff2c1555039b" containerName="payload-processing"
Sep 06 19:37:12.768887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.768811    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3558bcb-30b1-4990-9ed8-596cecfc9139" containerName="main"
Sep 06 19:37:12.768887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.768817    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3c6f539-5d94-4d16-a008-2dacfe6ac568" containerName="maas-api"
Sep 06 19:37:12.771643 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.771628    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"
Sep 06 19:37:12.773769 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.773747    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c31435-serving-cert\""
Sep 06 19:37:12.782775 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.782753    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"]
Sep 06 19:37:12.876540 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.876510    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2fc63c5-be2e-4253-997b-06ae60ed54be-maas-api-tls\") pod \"maas-api-e2e-api-a-c31435-847d5f4697-pxxzg\" (UID: \"a2fc63c5-be2e-4253-997b-06ae60ed54be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"
Sep 06 19:37:12.876728 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.876566    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9zdr\" (UniqueName: \"kubernetes.io/projected/a2fc63c5-be2e-4253-997b-06ae60ed54be-kube-api-access-w9zdr\") pod \"maas-api-e2e-api-a-c31435-847d5f4697-pxxzg\" (UID: \"a2fc63c5-be2e-4253-997b-06ae60ed54be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"
Sep 06 19:37:12.876728 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.876718    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2fc63c5-be2e-4253-997b-06ae60ed54be-openshift-service-ca\") pod \"maas-api-e2e-api-a-c31435-847d5f4697-pxxzg\" (UID: \"a2fc63c5-be2e-4253-997b-06ae60ed54be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"
Sep 06 19:37:12.876850 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.876763    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2fc63c5-be2e-4253-997b-06ae60ed54be-metrics-certs\") pod \"maas-api-e2e-api-a-c31435-847d5f4697-pxxzg\" (UID: \"a2fc63c5-be2e-4253-997b-06ae60ed54be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"
Sep 06 19:37:12.977731 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.977700    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2fc63c5-be2e-4253-997b-06ae60ed54be-openshift-service-ca\") pod \"maas-api-e2e-api-a-c31435-847d5f4697-pxxzg\" (UID: \"a2fc63c5-be2e-4253-997b-06ae60ed54be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"
Sep 06 19:37:12.977884 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.977823    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2fc63c5-be2e-4253-997b-06ae60ed54be-metrics-certs\") pod \"maas-api-e2e-api-a-c31435-847d5f4697-pxxzg\" (UID: \"a2fc63c5-be2e-4253-997b-06ae60ed54be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"
Sep 06 19:37:12.977884 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.977878    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2fc63c5-be2e-4253-997b-06ae60ed54be-maas-api-tls\") pod \"maas-api-e2e-api-a-c31435-847d5f4697-pxxzg\" (UID: \"a2fc63c5-be2e-4253-997b-06ae60ed54be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"
Sep 06 19:37:12.978005 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.977929    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9zdr\" (UniqueName: \"kubernetes.io/projected/a2fc63c5-be2e-4253-997b-06ae60ed54be-kube-api-access-w9zdr\") pod \"maas-api-e2e-api-a-c31435-847d5f4697-pxxzg\" (UID: \"a2fc63c5-be2e-4253-997b-06ae60ed54be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"
Sep 06 19:37:12.978067 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:37:12.978040    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-serving-cert: secret "maas-api-e2e-api-a-c31435-serving-cert" not found
Sep 06 19:37:12.978122 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:37:12.978109    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2fc63c5-be2e-4253-997b-06ae60ed54be-maas-api-tls podName:a2fc63c5-be2e-4253-997b-06ae60ed54be nodeName:}" failed. No retries permitted until 2026-09-06 19:37:13.478088731 +0000 UTC m=+1567.659270519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a2fc63c5-be2e-4253-997b-06ae60ed54be-maas-api-tls") pod "maas-api-e2e-api-a-c31435-847d5f4697-pxxzg" (UID: "a2fc63c5-be2e-4253-997b-06ae60ed54be") : secret "maas-api-e2e-api-a-c31435-serving-cert" not found
Sep 06 19:37:12.978303 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.978281    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2fc63c5-be2e-4253-997b-06ae60ed54be-openshift-service-ca\") pod \"maas-api-e2e-api-a-c31435-847d5f4697-pxxzg\" (UID: \"a2fc63c5-be2e-4253-997b-06ae60ed54be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"
Sep 06 19:37:12.980380 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.980361    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2fc63c5-be2e-4253-997b-06ae60ed54be-metrics-certs\") pod \"maas-api-e2e-api-a-c31435-847d5f4697-pxxzg\" (UID: \"a2fc63c5-be2e-4253-997b-06ae60ed54be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"
Sep 06 19:37:12.986537 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:12.986513    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9zdr\" (UniqueName: \"kubernetes.io/projected/a2fc63c5-be2e-4253-997b-06ae60ed54be-kube-api-access-w9zdr\") pod \"maas-api-e2e-api-a-c31435-847d5f4697-pxxzg\" (UID: \"a2fc63c5-be2e-4253-997b-06ae60ed54be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"
Sep 06 19:37:13.318742 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:13.318713    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr"]
Sep 06 19:37:13.322378 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:13.322362    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr"
Sep 06 19:37:13.325237 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:13.325214    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c31435\""
Sep 06 19:37:13.325359 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:13.325310    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c31435-dockercfg-7cnvr\""
Sep 06 19:37:13.331705 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:13.331685    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr"]
Sep 06 19:37:13.482533 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:13.482505    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2fc63c5-be2e-4253-997b-06ae60ed54be-maas-api-tls\") pod \"maas-api-e2e-api-a-c31435-847d5f4697-pxxzg\" (UID: \"a2fc63c5-be2e-4253-997b-06ae60ed54be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"
Sep 06 19:37:13.482738 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:13.482571    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2047fe90-22c3-493f-b441-3d33a0c40964-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr\" (UID: \"2047fe90-22c3-493f-b441-3d33a0c40964\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr"
Sep 06 19:37:13.482738 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:13.482644    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xn4h\" (UniqueName: \"kubernetes.io/projected/2047fe90-22c3-493f-b441-3d33a0c40964-kube-api-access-5xn4h\") pod \"payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr\" (UID: \"2047fe90-22c3-493f-b441-3d33a0c40964\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr"
Sep 06 19:37:13.484939 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:13.484917    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2fc63c5-be2e-4253-997b-06ae60ed54be-maas-api-tls\") pod \"maas-api-e2e-api-a-c31435-847d5f4697-pxxzg\" (UID: \"a2fc63c5-be2e-4253-997b-06ae60ed54be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"
Sep 06 19:37:13.583958 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:13.583870    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2047fe90-22c3-493f-b441-3d33a0c40964-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr\" (UID: \"2047fe90-22c3-493f-b441-3d33a0c40964\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr"
Sep 06 19:37:13.583958 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:13.583936    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xn4h\" (UniqueName: \"kubernetes.io/projected/2047fe90-22c3-493f-b441-3d33a0c40964-kube-api-access-5xn4h\") pod \"payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr\" (UID: \"2047fe90-22c3-493f-b441-3d33a0c40964\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr"
Sep 06 19:37:13.584559 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:13.584539    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2047fe90-22c3-493f-b441-3d33a0c40964-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr\" (UID: \"2047fe90-22c3-493f-b441-3d33a0c40964\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr"
Sep 06 19:37:13.591467 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:13.591440    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xn4h\" (UniqueName: \"kubernetes.io/projected/2047fe90-22c3-493f-b441-3d33a0c40964-kube-api-access-5xn4h\") pod \"payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr\" (UID: \"2047fe90-22c3-493f-b441-3d33a0c40964\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr"
Sep 06 19:37:13.633484 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:13.633460    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr"
Sep 06 19:37:13.685956 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:13.685655    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"
Sep 06 19:37:13.866755 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:13.866676    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7722-2817d1"]
Sep 06 19:37:13.870412 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:13.870396    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7722-2817d1"
Sep 06 19:37:13.872782 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:13.872758    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zsskm\""
Sep 06 19:37:13.876132 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:13.876110    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7722-2817d1"]
Sep 06 19:37:13.903683 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:13.903663    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr"]
Sep 06 19:37:13.904947 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:37:13.904921    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2047fe90_22c3_493f_b441_3d33a0c40964.slice/crio-cd464e0d32252aafea141827767f351c268c9ad2faf0a812e0abfb1f4aa1c279 WatchSource:0}: Error finding container cd464e0d32252aafea141827767f351c268c9ad2faf0a812e0abfb1f4aa1c279: Status 404 returned error can't find the container with id cd464e0d32252aafea141827767f351c268c9ad2faf0a812e0abfb1f4aa1c279
Sep 06 19:37:13.966627 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:13.966590    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"]
Sep 06 19:37:13.968560 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:37:13.968529    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2fc63c5_be2e_4253_997b_06ae60ed54be.slice/crio-f10ce7e73f2384bab3aa7fb4365f0d121d3f851bede56b507ad6b95a5044a7ce WatchSource:0}: Error finding container f10ce7e73f2384bab3aa7fb4365f0d121d3f851bede56b507ad6b95a5044a7ce: Status 404 returned error can't find the container with id f10ce7e73f2384bab3aa7fb4365f0d121d3f851bede56b507ad6b95a5044a7ce
Sep 06 19:37:13.989006 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:13.988983    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klpm2\" (UniqueName: \"kubernetes.io/projected/19dcebd4-e606-424b-99c7-4ffb62d8f777-kube-api-access-klpm2\") pod \"mt-curl-7722-2817d1\" (UID: \"19dcebd4-e606-424b-99c7-4ffb62d8f777\") " pod="opendatahub/mt-curl-7722-2817d1"
Sep 06 19:37:14.089586 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:14.089545    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klpm2\" (UniqueName: \"kubernetes.io/projected/19dcebd4-e606-424b-99c7-4ffb62d8f777-kube-api-access-klpm2\") pod \"mt-curl-7722-2817d1\" (UID: \"19dcebd4-e606-424b-99c7-4ffb62d8f777\") " pod="opendatahub/mt-curl-7722-2817d1"
Sep 06 19:37:14.096856 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:14.096832    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klpm2\" (UniqueName: \"kubernetes.io/projected/19dcebd4-e606-424b-99c7-4ffb62d8f777-kube-api-access-klpm2\") pod \"mt-curl-7722-2817d1\" (UID: \"19dcebd4-e606-424b-99c7-4ffb62d8f777\") " pod="opendatahub/mt-curl-7722-2817d1"
Sep 06 19:37:14.121306 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:14.121232    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg" event={"ID":"a2fc63c5-be2e-4253-997b-06ae60ed54be","Type":"ContainerStarted","Data":"f10ce7e73f2384bab3aa7fb4365f0d121d3f851bede56b507ad6b95a5044a7ce"}
Sep 06 19:37:14.122894 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:14.122867    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr" event={"ID":"2047fe90-22c3-493f-b441-3d33a0c40964","Type":"ContainerStarted","Data":"9bb48e64b5032d5a9885e6e34ef2e32cb1d57cb71681bd2840baf1e49621aaf3"}
Sep 06 19:37:14.123024 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:14.122918    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr" event={"ID":"2047fe90-22c3-493f-b441-3d33a0c40964","Type":"ContainerStarted","Data":"cd464e0d32252aafea141827767f351c268c9ad2faf0a812e0abfb1f4aa1c279"}
Sep 06 19:37:14.123024 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:14.122987    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr"
Sep 06 19:37:14.137935 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:14.137895    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr" podStartSLOduration=1.13788278 podStartE2EDuration="1.13788278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:37:14.136289169 +0000 UTC m=+1568.317470975" watchObservedRunningTime="2026-09-06 19:37:14.13788278 +0000 UTC m=+1568.319064651"
Sep 06 19:37:14.181357 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:14.181334    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7722-2817d1"
Sep 06 19:37:14.454534 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:14.454501    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7722-2817d1"]
Sep 06 19:37:14.454757 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:37:14.454735    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19dcebd4_e606_424b_99c7_4ffb62d8f777.slice/crio-baa6071733a271118c3489585af4d1414318506a42f7162abf11d5f5d0bf808f WatchSource:0}: Error finding container baa6071733a271118c3489585af4d1414318506a42f7162abf11d5f5d0bf808f: Status 404 returned error can't find the container with id baa6071733a271118c3489585af4d1414318506a42f7162abf11d5f5d0bf808f
Sep 06 19:37:15.130409 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:15.130378    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg" event={"ID":"a2fc63c5-be2e-4253-997b-06ae60ed54be","Type":"ContainerStarted","Data":"02bb6acf15aed9c0f44f13f54640c3e9b61456631c3f40b3be837e036ed9b412"}
Sep 06 19:37:15.130786 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:15.130515    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"
Sep 06 19:37:15.131569 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:15.131547    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7722-2817d1" event={"ID":"19dcebd4-e606-424b-99c7-4ffb62d8f777","Type":"ContainerStarted","Data":"baa6071733a271118c3489585af4d1414318506a42f7162abf11d5f5d0bf808f"}
Sep 06 19:37:15.147017 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:15.146973    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg" podStartSLOduration=2.75906982 podStartE2EDuration="3.146955105s" totalImagesPullingTime="387.885285ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:37:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:37:15.146064778 +0000 UTC m=+1569.327246585" watchObservedRunningTime="2026-09-06 19:37:15.146955105 +0000 UTC m=+1569.328136914"
Sep 06 19:37:16.139154 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:16.139119    2566 generic.go:356] "Generic (PLEG): container finished" podID="19dcebd4-e606-424b-99c7-4ffb62d8f777" containerID="89cf465bd1e280bd3711ccc09c490ecdfa7c477fe95ff3dfd686b2f91225699e" exitCode=0
Sep 06 19:37:16.139536 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:16.139178    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7722-2817d1" event={"ID":"19dcebd4-e606-424b-99c7-4ffb62d8f777","Type":"ContainerDied","Data":"89cf465bd1e280bd3711ccc09c490ecdfa7c477fe95ff3dfd686b2f91225699e"}
Sep 06 19:37:17.280968 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:17.280945    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7722-2817d1"
Sep 06 19:37:17.309779 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:17.309756    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7722-2817d1_19dcebd4-e606-424b-99c7-4ffb62d8f777/mt-curl-7722-2817d1/0.log"
Sep 06 19:37:17.331412 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:17.331390    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7722-2817d1"]
Sep 06 19:37:17.334759 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:17.334739    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7722-2817d1"]
Sep 06 19:37:17.424998 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:17.424876    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klpm2\" (UniqueName: \"kubernetes.io/projected/19dcebd4-e606-424b-99c7-4ffb62d8f777-kube-api-access-klpm2\") pod \"19dcebd4-e606-424b-99c7-4ffb62d8f777\" (UID: \"19dcebd4-e606-424b-99c7-4ffb62d8f777\") "
Sep 06 19:37:17.426992 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:17.426968    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19dcebd4-e606-424b-99c7-4ffb62d8f777-kube-api-access-klpm2" (OuterVolumeSpecName: "kube-api-access-klpm2") pod "19dcebd4-e606-424b-99c7-4ffb62d8f777" (UID: "19dcebd4-e606-424b-99c7-4ffb62d8f777"). InnerVolumeSpecName "kube-api-access-klpm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:37:17.525907 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:17.525874    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klpm2\" (UniqueName: \"kubernetes.io/projected/19dcebd4-e606-424b-99c7-4ffb62d8f777-kube-api-access-klpm2\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:37:18.153114 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:18.153083    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7722-2817d1"
Sep 06 19:37:18.153282 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:18.153125    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baa6071733a271118c3489585af4d1414318506a42f7162abf11d5f5d0bf808f"
Sep 06 19:37:18.168221 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:37:18.168191    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7722-2817d1\" is forbidden: User \"system:node:ip-10-0-129-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-252.ec2.internal' and this object" podUID="19dcebd4-e606-424b-99c7-4ffb62d8f777" pod="opendatahub/mt-curl-7722-2817d1"
Sep 06 19:37:18.311955 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:18.311923    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19dcebd4-e606-424b-99c7-4ffb62d8f777" path="/var/lib/kubelet/pods/19dcebd4-e606-424b-99c7-4ffb62d8f777/volumes"
Sep 06 19:37:18.696791 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:18.696760    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"]
Sep 06 19:37:21.144535 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:21.144509    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"
Sep 06 19:37:25.134828 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:25.134787    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr"
Sep 06 19:37:30.141817 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:30.141779    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:37:31.939006 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:31.938974    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"]
Sep 06 19:37:31.939427 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:31.939413    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19dcebd4-e606-424b-99c7-4ffb62d8f777" containerName="mt-curl-7722-2817d1"
Sep 06 19:37:31.939471 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:31.939430    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="19dcebd4-e606-424b-99c7-4ffb62d8f777" containerName="mt-curl-7722-2817d1"
Sep 06 19:37:31.939532 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:31.939523    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="19dcebd4-e606-424b-99c7-4ffb62d8f777" containerName="mt-curl-7722-2817d1"
Sep 06 19:37:31.942894 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:31.942874    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:31.945095 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:31.945074    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e5ef25\"/\"sub-test-model-e5ef25-kserve-self-signed-certs\""
Sep 06 19:37:31.950940 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:31.950919    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"]
Sep 06 19:37:31.957650 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:31.957623    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-model-cache\") pod \"sub-test-model-e5ef25-kserve-649cf654c-55vfd\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:31.957780 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:31.957761    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-dshm\") pod \"sub-test-model-e5ef25-kserve-649cf654c-55vfd\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:31.957850 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:31.957809    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-tmp-dir\") pod \"sub-test-model-e5ef25-kserve-649cf654c-55vfd\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:31.957903 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:31.957883    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-home\") pod \"sub-test-model-e5ef25-kserve-649cf654c-55vfd\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:31.957961 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:31.957939    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-tls-certs\") pod \"sub-test-model-e5ef25-kserve-649cf654c-55vfd\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:31.958018 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:31.957975    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lvnw\" (UniqueName: \"kubernetes.io/projected/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-kube-api-access-4lvnw\") pod \"sub-test-model-e5ef25-kserve-649cf654c-55vfd\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:32.058555 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:32.058525    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-model-cache\") pod \"sub-test-model-e5ef25-kserve-649cf654c-55vfd\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:32.058732 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:32.058567    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-dshm\") pod \"sub-test-model-e5ef25-kserve-649cf654c-55vfd\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:32.058732 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:32.058696    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-tmp-dir\") pod \"sub-test-model-e5ef25-kserve-649cf654c-55vfd\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:32.058854 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:32.058737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-home\") pod \"sub-test-model-e5ef25-kserve-649cf654c-55vfd\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:32.058854 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:32.058775    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-tls-certs\") pod \"sub-test-model-e5ef25-kserve-649cf654c-55vfd\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:32.058854 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:32.058802    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lvnw\" (UniqueName: \"kubernetes.io/projected/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-kube-api-access-4lvnw\") pod \"sub-test-model-e5ef25-kserve-649cf654c-55vfd\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:32.059016 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:32.058864    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-model-cache\") pod \"sub-test-model-e5ef25-kserve-649cf654c-55vfd\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:32.059074 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:32.059043    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-tmp-dir\") pod \"sub-test-model-e5ef25-kserve-649cf654c-55vfd\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:32.059074 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:32.059055    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-home\") pod \"sub-test-model-e5ef25-kserve-649cf654c-55vfd\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:32.060948 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:32.060920    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-dshm\") pod \"sub-test-model-e5ef25-kserve-649cf654c-55vfd\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:32.061322 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:32.061303    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-tls-certs\") pod \"sub-test-model-e5ef25-kserve-649cf654c-55vfd\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:32.066665 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:32.066645    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lvnw\" (UniqueName: \"kubernetes.io/projected/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-kube-api-access-4lvnw\") pod \"sub-test-model-e5ef25-kserve-649cf654c-55vfd\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:32.252598 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:32.252567    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:32.523781 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:32.523753    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"]
Sep 06 19:37:32.524854 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:37:32.524833    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb9fed6a_8027_4e3d_9305_a96426d7c0aa.slice/crio-27f09df507ed153f2819b86dc169bd73e2445ec75f9ab915dfd88551542d3150 WatchSource:0}: Error finding container 27f09df507ed153f2819b86dc169bd73e2445ec75f9ab915dfd88551542d3150: Status 404 returned error can't find the container with id 27f09df507ed153f2819b86dc169bd73e2445ec75f9ab915dfd88551542d3150
Sep 06 19:37:33.243422 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:33.243387    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd" event={"ID":"bb9fed6a-8027-4e3d-9305-a96426d7c0aa","Type":"ContainerStarted","Data":"c7973a6b357f3f6a5c02afd9ee9e8839b1b4b14e697d4dc9d97ed4499e8a57ad"}
Sep 06 19:37:33.243422 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:33.243429    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd" event={"ID":"bb9fed6a-8027-4e3d-9305-a96426d7c0aa","Type":"ContainerStarted","Data":"27f09df507ed153f2819b86dc169bd73e2445ec75f9ab915dfd88551542d3150"}
Sep 06 19:37:33.258572 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:33.258521    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd" podStartSLOduration=2.258502203 podStartE2EDuration="2.258502203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:37:33.258022315 +0000 UTC m=+1587.439204123" watchObservedRunningTime="2026-09-06 19:37:33.258502203 +0000 UTC m=+1587.439684010"
Sep 06 19:37:33.716465 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:33.716432    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn" podUID="b118acb5-c68d-4a81-8698-44c4e81286d1" containerName="main" containerID="cri-o://06551203e4d7cda5cf66b224e2bb865233f67ab0c267a50a19610f6d8c2f0ce6" gracePeriod=45
Sep 06 19:37:33.971919 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:33.971840    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:37:34.077042 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.077015    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8f7qf\" (UniqueName: \"kubernetes.io/projected/b118acb5-c68d-4a81-8698-44c4e81286d1-kube-api-access-8f7qf\") pod \"b118acb5-c68d-4a81-8698-44c4e81286d1\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") "
Sep 06 19:37:34.077186 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.077055    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-dshm\") pod \"b118acb5-c68d-4a81-8698-44c4e81286d1\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") "
Sep 06 19:37:34.077186 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.077104    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-model-cache\") pod \"b118acb5-c68d-4a81-8698-44c4e81286d1\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") "
Sep 06 19:37:34.077276 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.077194    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b118acb5-c68d-4a81-8698-44c4e81286d1-tls-certs\") pod \"b118acb5-c68d-4a81-8698-44c4e81286d1\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") "
Sep 06 19:37:34.077276 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.077250    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-home\") pod \"b118acb5-c68d-4a81-8698-44c4e81286d1\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") "
Sep 06 19:37:34.077354 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.077295    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "b118acb5-c68d-4a81-8698-44c4e81286d1" (UID: "b118acb5-c68d-4a81-8698-44c4e81286d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:37:34.077354 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.077311    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-tmp-dir\") pod \"b118acb5-c68d-4a81-8698-44c4e81286d1\" (UID: \"b118acb5-c68d-4a81-8698-44c4e81286d1\") "
Sep 06 19:37:34.077518 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.077487    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-home" (OuterVolumeSpecName: "home") pod "b118acb5-c68d-4a81-8698-44c4e81286d1" (UID: "b118acb5-c68d-4a81-8698-44c4e81286d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:37:34.077649 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.077624    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b118acb5-c68d-4a81-8698-44c4e81286d1" (UID: "b118acb5-c68d-4a81-8698-44c4e81286d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:37:34.077742 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.077725    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-model-cache\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:37:34.077803 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.077745    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-home\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:37:34.077803 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.077757    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-tmp-dir\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:37:34.079510 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.079482    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b118acb5-c68d-4a81-8698-44c4e81286d1-kube-api-access-8f7qf" (OuterVolumeSpecName: "kube-api-access-8f7qf") pod "b118acb5-c68d-4a81-8698-44c4e81286d1" (UID: "b118acb5-c68d-4a81-8698-44c4e81286d1"). InnerVolumeSpecName "kube-api-access-8f7qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:37:34.079625 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.079529    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b118acb5-c68d-4a81-8698-44c4e81286d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b118acb5-c68d-4a81-8698-44c4e81286d1" (UID: "b118acb5-c68d-4a81-8698-44c4e81286d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:37:34.079625 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.079540    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-dshm" (OuterVolumeSpecName: "dshm") pod "b118acb5-c68d-4a81-8698-44c4e81286d1" (UID: "b118acb5-c68d-4a81-8698-44c4e81286d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:37:34.178193 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.178167    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b118acb5-c68d-4a81-8698-44c4e81286d1-tls-certs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:37:34.178193 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.178189    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f7qf\" (UniqueName: \"kubernetes.io/projected/b118acb5-c68d-4a81-8698-44c4e81286d1-kube-api-access-8f7qf\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:37:34.178331 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.178198    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b118acb5-c68d-4a81-8698-44c4e81286d1-dshm\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:37:34.249823 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.249752    2566 generic.go:356] "Generic (PLEG): container finished" podID="b118acb5-c68d-4a81-8698-44c4e81286d1" containerID="06551203e4d7cda5cf66b224e2bb865233f67ab0c267a50a19610f6d8c2f0ce6" exitCode=0
Sep 06 19:37:34.249823 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.249803    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"
Sep 06 19:37:34.250201 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.249837    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn" event={"ID":"b118acb5-c68d-4a81-8698-44c4e81286d1","Type":"ContainerDied","Data":"06551203e4d7cda5cf66b224e2bb865233f67ab0c267a50a19610f6d8c2f0ce6"}
Sep 06 19:37:34.250201 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.249884    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn" event={"ID":"b118acb5-c68d-4a81-8698-44c4e81286d1","Type":"ContainerDied","Data":"ac815859a728863d419a75811e2c2deb34a79f8fe68c39909cc095c75d3097bc"}
Sep 06 19:37:34.250201 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.249901    2566 scope.go:117] "RemoveContainer" containerID="06551203e4d7cda5cf66b224e2bb865233f67ab0c267a50a19610f6d8c2f0ce6"
Sep 06 19:37:34.260089 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.260071    2566 scope.go:117] "RemoveContainer" containerID="06551203e4d7cda5cf66b224e2bb865233f67ab0c267a50a19610f6d8c2f0ce6"
Sep 06 19:37:34.260370 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:37:34.260348    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06551203e4d7cda5cf66b224e2bb865233f67ab0c267a50a19610f6d8c2f0ce6\": container with ID starting with 06551203e4d7cda5cf66b224e2bb865233f67ab0c267a50a19610f6d8c2f0ce6 not found: ID does not exist" containerID="06551203e4d7cda5cf66b224e2bb865233f67ab0c267a50a19610f6d8c2f0ce6"
Sep 06 19:37:34.260421 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.260379    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06551203e4d7cda5cf66b224e2bb865233f67ab0c267a50a19610f6d8c2f0ce6"} err="failed to get container status \"06551203e4d7cda5cf66b224e2bb865233f67ab0c267a50a19610f6d8c2f0ce6\": rpc error: code = NotFound desc = could not find container \"06551203e4d7cda5cf66b224e2bb865233f67ab0c267a50a19610f6d8c2f0ce6\": container with ID starting with 06551203e4d7cda5cf66b224e2bb865233f67ab0c267a50a19610f6d8c2f0ce6 not found: ID does not exist"
Sep 06 19:37:34.270813 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.270793    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"]
Sep 06 19:37:34.274507 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.274486    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5ef25/auth-test-model-e5ef25-kserve-5bbcc9876c-78wnn"]
Sep 06 19:37:34.310915 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:34.310890    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b118acb5-c68d-4a81-8698-44c4e81286d1" path="/var/lib/kubelet/pods/b118acb5-c68d-4a81-8698-44c4e81286d1/volumes"
Sep 06 19:37:41.895723 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:41.895689    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"]
Sep 06 19:37:41.896211 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:41.896147    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b118acb5-c68d-4a81-8698-44c4e81286d1" containerName="main"
Sep 06 19:37:41.896211 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:41.896162    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b118acb5-c68d-4a81-8698-44c4e81286d1" containerName="main"
Sep 06 19:37:41.896331 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:41.896223    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b118acb5-c68d-4a81-8698-44c4e81286d1" containerName="main"
Sep 06 19:37:41.899289 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:41.899270    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"
Sep 06 19:37:41.901563 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:41.901543    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-49a2d2-serving-cert\""
Sep 06 19:37:41.910913 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:41.910887    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"]
Sep 06 19:37:41.944481 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:41.944454    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-openshift-service-ca\") pod \"maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd\" (UID: \"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"
Sep 06 19:37:41.944629 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:41.944485    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-maas-api-tls\") pod \"maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd\" (UID: \"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"
Sep 06 19:37:41.944629 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:41.944522    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-metrics-certs\") pod \"maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd\" (UID: \"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"
Sep 06 19:37:41.944629 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:41.944581    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9k4x\" (UniqueName: \"kubernetes.io/projected/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-kube-api-access-w9k4x\") pod \"maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd\" (UID: \"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"
Sep 06 19:37:42.045160 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.045125    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-openshift-service-ca\") pod \"maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd\" (UID: \"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"
Sep 06 19:37:42.045330 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.045176    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-maas-api-tls\") pod \"maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd\" (UID: \"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"
Sep 06 19:37:42.045330 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.045235    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-metrics-certs\") pod \"maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd\" (UID: \"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"
Sep 06 19:37:42.045330 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.045278    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9k4x\" (UniqueName: \"kubernetes.io/projected/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-kube-api-access-w9k4x\") pod \"maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd\" (UID: \"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"
Sep 06 19:37:42.045330 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:37:42.045304    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-serving-cert: secret "maas-api-e2e-api-b-49a2d2-serving-cert" not found
Sep 06 19:37:42.045535 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:37:42.045371    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-maas-api-tls podName:48c4f9d2-3e07-4f6c-af7a-7c796a894bf8 nodeName:}" failed. No retries permitted until 2026-09-06 19:37:42.54534943 +0000 UTC m=+1596.726531233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-maas-api-tls") pod "maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd" (UID: "48c4f9d2-3e07-4f6c-af7a-7c796a894bf8") : secret "maas-api-e2e-api-b-49a2d2-serving-cert" not found
Sep 06 19:37:42.049924 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.045884    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-openshift-service-ca\") pod \"maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd\" (UID: \"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"
Sep 06 19:37:42.050447 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.050425    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-metrics-certs\") pod \"maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd\" (UID: \"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"
Sep 06 19:37:42.053510 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.053491    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9k4x\" (UniqueName: \"kubernetes.io/projected/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-kube-api-access-w9k4x\") pod \"maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd\" (UID: \"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"
Sep 06 19:37:42.253334 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.253293    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:42.253334 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.253340    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:42.265851 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.265826    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:42.314900 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.314872    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:37:42.533204 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.533127    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp"]
Sep 06 19:37:42.536874 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.536846    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp"
Sep 06 19:37:42.539255 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.539227    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-49a2d2-dockercfg-ff5fc\""
Sep 06 19:37:42.539392 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.539263    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-49a2d2\""
Sep 06 19:37:42.548183 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.548162    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp"]
Sep 06 19:37:42.550569 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.550547    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-maas-api-tls\") pod \"maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd\" (UID: \"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"
Sep 06 19:37:42.553167 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.553149    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-maas-api-tls\") pod \"maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd\" (UID: \"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"
Sep 06 19:37:42.651138 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.651109    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv4sw\" (UniqueName: \"kubernetes.io/projected/ae2de6c6-8885-486d-bac3-92bef9a3bfc0-kube-api-access-zv4sw\") pod \"payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp\" (UID: \"ae2de6c6-8885-486d-bac3-92bef9a3bfc0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp"
Sep 06 19:37:42.651285 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.651175    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae2de6c6-8885-486d-bac3-92bef9a3bfc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp\" (UID: \"ae2de6c6-8885-486d-bac3-92bef9a3bfc0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp"
Sep 06 19:37:42.751907 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.751867    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae2de6c6-8885-486d-bac3-92bef9a3bfc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp\" (UID: \"ae2de6c6-8885-486d-bac3-92bef9a3bfc0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp"
Sep 06 19:37:42.752047 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.752002    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv4sw\" (UniqueName: \"kubernetes.io/projected/ae2de6c6-8885-486d-bac3-92bef9a3bfc0-kube-api-access-zv4sw\") pod \"payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp\" (UID: \"ae2de6c6-8885-486d-bac3-92bef9a3bfc0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp"
Sep 06 19:37:42.752526 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.752508    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae2de6c6-8885-486d-bac3-92bef9a3bfc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp\" (UID: \"ae2de6c6-8885-486d-bac3-92bef9a3bfc0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp"
Sep 06 19:37:42.759511 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.759492    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv4sw\" (UniqueName: \"kubernetes.io/projected/ae2de6c6-8885-486d-bac3-92bef9a3bfc0-kube-api-access-zv4sw\") pod \"payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp\" (UID: \"ae2de6c6-8885-486d-bac3-92bef9a3bfc0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp"
Sep 06 19:37:42.813153 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.813088    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"
Sep 06 19:37:42.849429 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:42.849387    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp"
Sep 06 19:37:43.088455 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:43.088421    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"]
Sep 06 19:37:43.089631 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:37:43.089583    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c4f9d2_3e07_4f6c_af7a_7c796a894bf8.slice/crio-19af85d08035fe799da70bff09bc6edf05773bb7a4814b3f5874d475b4c08d58 WatchSource:0}: Error finding container 19af85d08035fe799da70bff09bc6edf05773bb7a4814b3f5874d475b4c08d58: Status 404 returned error can't find the container with id 19af85d08035fe799da70bff09bc6edf05773bb7a4814b3f5874d475b4c08d58
Sep 06 19:37:43.133202 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:43.133174    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp"]
Sep 06 19:37:43.134145 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:37:43.134117    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2de6c6_8885_486d_bac3_92bef9a3bfc0.slice/crio-1198895e37ce43b2abb2e44cf83383b0bf9544f82f3be836f6ca2127677f0ae9 WatchSource:0}: Error finding container 1198895e37ce43b2abb2e44cf83383b0bf9544f82f3be836f6ca2127677f0ae9: Status 404 returned error can't find the container with id 1198895e37ce43b2abb2e44cf83383b0bf9544f82f3be836f6ca2127677f0ae9
Sep 06 19:37:43.310935 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:43.310906    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp" event={"ID":"ae2de6c6-8885-486d-bac3-92bef9a3bfc0","Type":"ContainerStarted","Data":"9eafa43d97083fdd0ce6d425cad6024ee0390cebcd3a7af4e51b4135dfb80035"}
Sep 06 19:37:43.311117 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:43.310956    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp" event={"ID":"ae2de6c6-8885-486d-bac3-92bef9a3bfc0","Type":"ContainerStarted","Data":"1198895e37ce43b2abb2e44cf83383b0bf9544f82f3be836f6ca2127677f0ae9"}
Sep 06 19:37:43.311117 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:43.311106    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp"
Sep 06 19:37:43.312230 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:43.312205    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd" event={"ID":"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8","Type":"ContainerStarted","Data":"19af85d08035fe799da70bff09bc6edf05773bb7a4814b3f5874d475b4c08d58"}
Sep 06 19:37:43.326571 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:43.326527    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp" podStartSLOduration=1.326513029 podStartE2EDuration="1.326513029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:37:43.324766684 +0000 UTC m=+1597.505948490" watchObservedRunningTime="2026-09-06 19:37:43.326513029 +0000 UTC m=+1597.507694834"
Sep 06 19:37:44.319868 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:44.319834    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd" event={"ID":"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8","Type":"ContainerStarted","Data":"ac3c853e782f633e3ba25dd05cb9f94b4e7d4a49b808905afe0dc16ed99df184"}
Sep 06 19:37:44.320283 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:44.320048    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"
Sep 06 19:37:44.337247 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:44.337201    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd" podStartSLOduration=3.049278233 podStartE2EDuration="3.337187171s" totalImagesPullingTime="287.908938ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:37:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:37:44.33553803 +0000 UTC m=+1598.516719840" watchObservedRunningTime="2026-09-06 19:37:44.337187171 +0000 UTC m=+1598.518368976"
Sep 06 19:37:46.651511 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:46.651475    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:37:50.331550 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:50.331519    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"
Sep 06 19:37:54.322319 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:37:54.322290    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp"
Sep 06 19:38:08.851914 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:08.851872    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:38:22.647179 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:22.647146    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:38:24.286559 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.286524    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"]
Sep 06 19:38:24.287057 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.286770    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg" podUID="a2fc63c5-be2e-4253-997b-06ae60ed54be" containerName="maas-api" containerID="cri-o://02bb6acf15aed9c0f44f13f54640c3e9b61456631c3f40b3be837e036ed9b412" gracePeriod=30
Sep 06 19:38:24.338981 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.338912    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr"]
Sep 06 19:38:24.339427 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.339388    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr" podUID="2047fe90-22c3-493f-b441-3d33a0c40964" containerName="payload-pre-processing" containerID="cri-o://9bb48e64b5032d5a9885e6e34ef2e32cb1d57cb71681bd2840baf1e49621aaf3" gracePeriod=30
Sep 06 19:38:24.553024 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.552938    2566 generic.go:356] "Generic (PLEG): container finished" podID="a2fc63c5-be2e-4253-997b-06ae60ed54be" containerID="02bb6acf15aed9c0f44f13f54640c3e9b61456631c3f40b3be837e036ed9b412" exitCode=0
Sep 06 19:38:24.553172 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.553020    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg" event={"ID":"a2fc63c5-be2e-4253-997b-06ae60ed54be","Type":"ContainerDied","Data":"02bb6acf15aed9c0f44f13f54640c3e9b61456631c3f40b3be837e036ed9b412"}
Sep 06 19:38:24.555094 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.555067    2566 generic.go:356] "Generic (PLEG): container finished" podID="2047fe90-22c3-493f-b441-3d33a0c40964" containerID="9bb48e64b5032d5a9885e6e34ef2e32cb1d57cb71681bd2840baf1e49621aaf3" exitCode=0
Sep 06 19:38:24.555210 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.555140    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr" event={"ID":"2047fe90-22c3-493f-b441-3d33a0c40964","Type":"ContainerDied","Data":"9bb48e64b5032d5a9885e6e34ef2e32cb1d57cb71681bd2840baf1e49621aaf3"}
Sep 06 19:38:24.603789 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.603770    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"
Sep 06 19:38:24.606923 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.606898    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr"
Sep 06 19:38:24.645276 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.645237    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2fc63c5-be2e-4253-997b-06ae60ed54be-metrics-certs\") pod \"a2fc63c5-be2e-4253-997b-06ae60ed54be\" (UID: \"a2fc63c5-be2e-4253-997b-06ae60ed54be\") "
Sep 06 19:38:24.645527 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.645502    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2fc63c5-be2e-4253-997b-06ae60ed54be-openshift-service-ca\") pod \"a2fc63c5-be2e-4253-997b-06ae60ed54be\" (UID: \"a2fc63c5-be2e-4253-997b-06ae60ed54be\") "
Sep 06 19:38:24.645582 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.645552    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2047fe90-22c3-493f-b441-3d33a0c40964-plugins-config-volume\") pod \"2047fe90-22c3-493f-b441-3d33a0c40964\" (UID: \"2047fe90-22c3-493f-b441-3d33a0c40964\") "
Sep 06 19:38:24.645671 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.645587    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9zdr\" (UniqueName: \"kubernetes.io/projected/a2fc63c5-be2e-4253-997b-06ae60ed54be-kube-api-access-w9zdr\") pod \"a2fc63c5-be2e-4253-997b-06ae60ed54be\" (UID: \"a2fc63c5-be2e-4253-997b-06ae60ed54be\") "
Sep 06 19:38:24.645804 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.645782    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xn4h\" (UniqueName: \"kubernetes.io/projected/2047fe90-22c3-493f-b441-3d33a0c40964-kube-api-access-5xn4h\") pod \"2047fe90-22c3-493f-b441-3d33a0c40964\" (UID: \"2047fe90-22c3-493f-b441-3d33a0c40964\") "
Sep 06 19:38:24.645889 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.645833    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2fc63c5-be2e-4253-997b-06ae60ed54be-maas-api-tls\") pod \"a2fc63c5-be2e-4253-997b-06ae60ed54be\" (UID: \"a2fc63c5-be2e-4253-997b-06ae60ed54be\") "
Sep 06 19:38:24.645945 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.645881    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2fc63c5-be2e-4253-997b-06ae60ed54be-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a2fc63c5-be2e-4253-997b-06ae60ed54be" (UID: "a2fc63c5-be2e-4253-997b-06ae60ed54be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:38:24.646033 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.646005    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2047fe90-22c3-493f-b441-3d33a0c40964-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2047fe90-22c3-493f-b441-3d33a0c40964" (UID: "2047fe90-22c3-493f-b441-3d33a0c40964"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:38:24.646208 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.646189    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2fc63c5-be2e-4253-997b-06ae60ed54be-openshift-service-ca\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:38:24.646317 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.646303    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2047fe90-22c3-493f-b441-3d33a0c40964-plugins-config-volume\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:38:24.648315 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.648267    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2047fe90-22c3-493f-b441-3d33a0c40964-kube-api-access-5xn4h" (OuterVolumeSpecName: "kube-api-access-5xn4h") pod "2047fe90-22c3-493f-b441-3d33a0c40964" (UID: "2047fe90-22c3-493f-b441-3d33a0c40964"). InnerVolumeSpecName "kube-api-access-5xn4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:38:24.648437 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.648358    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2fc63c5-be2e-4253-997b-06ae60ed54be-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a2fc63c5-be2e-4253-997b-06ae60ed54be" (UID: "a2fc63c5-be2e-4253-997b-06ae60ed54be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:38:24.648540 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.648515    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2fc63c5-be2e-4253-997b-06ae60ed54be-kube-api-access-w9zdr" (OuterVolumeSpecName: "kube-api-access-w9zdr") pod "a2fc63c5-be2e-4253-997b-06ae60ed54be" (UID: "a2fc63c5-be2e-4253-997b-06ae60ed54be"). InnerVolumeSpecName "kube-api-access-w9zdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:38:24.648968 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.648927    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2fc63c5-be2e-4253-997b-06ae60ed54be-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a2fc63c5-be2e-4253-997b-06ae60ed54be" (UID: "a2fc63c5-be2e-4253-997b-06ae60ed54be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:38:24.747792 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.747766    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2fc63c5-be2e-4253-997b-06ae60ed54be-metrics-certs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:38:24.747792 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.747789    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9zdr\" (UniqueName: \"kubernetes.io/projected/a2fc63c5-be2e-4253-997b-06ae60ed54be-kube-api-access-w9zdr\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:38:24.747952 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.747799    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xn4h\" (UniqueName: \"kubernetes.io/projected/2047fe90-22c3-493f-b441-3d33a0c40964-kube-api-access-5xn4h\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:38:24.747952 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:24.747808    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2fc63c5-be2e-4253-997b-06ae60ed54be-maas-api-tls\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:38:25.562361 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:25.562329    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"
Sep 06 19:38:25.562361 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:25.562352    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg" event={"ID":"a2fc63c5-be2e-4253-997b-06ae60ed54be","Type":"ContainerDied","Data":"f10ce7e73f2384bab3aa7fb4365f0d121d3f851bede56b507ad6b95a5044a7ce"}
Sep 06 19:38:25.562895 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:25.562417    2566 scope.go:117] "RemoveContainer" containerID="02bb6acf15aed9c0f44f13f54640c3e9b61456631c3f40b3be837e036ed9b412"
Sep 06 19:38:25.563909 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:25.563839    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr" event={"ID":"2047fe90-22c3-493f-b441-3d33a0c40964","Type":"ContainerDied","Data":"cd464e0d32252aafea141827767f351c268c9ad2faf0a812e0abfb1f4aa1c279"}
Sep 06 19:38:25.563909 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:25.563899    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr"
Sep 06 19:38:25.573033 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:25.573005    2566 scope.go:117] "RemoveContainer" containerID="9bb48e64b5032d5a9885e6e34ef2e32cb1d57cb71681bd2840baf1e49621aaf3"
Sep 06 19:38:25.585934 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:25.585913    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"]
Sep 06 19:38:25.589452 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:25.589428    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c31435-847d5f4697-pxxzg"]
Sep 06 19:38:25.607403 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:25.607380    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr"]
Sep 06 19:38:25.611134 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:25.611114    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c31435-6d49dbbb54-xsppr"]
Sep 06 19:38:26.312141 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:26.312107    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2047fe90-22c3-493f-b441-3d33a0c40964" path="/var/lib/kubelet/pods/2047fe90-22c3-493f-b441-3d33a0c40964/volumes"
Sep 06 19:38:26.312511 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:26.312497    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2fc63c5-be2e-4253-997b-06ae60ed54be" path="/var/lib/kubelet/pods/a2fc63c5-be2e-4253-997b-06ae60ed54be/volumes"
Sep 06 19:38:35.891459 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:35.891426    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7722-3057c9"]
Sep 06 19:38:35.892943 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:35.892916    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2047fe90-22c3-493f-b441-3d33a0c40964" containerName="payload-pre-processing"
Sep 06 19:38:35.893094 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:35.893082    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2047fe90-22c3-493f-b441-3d33a0c40964" containerName="payload-pre-processing"
Sep 06 19:38:35.893220 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:35.893209    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2fc63c5-be2e-4253-997b-06ae60ed54be" containerName="maas-api"
Sep 06 19:38:35.893291 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:35.893283    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2fc63c5-be2e-4253-997b-06ae60ed54be" containerName="maas-api"
Sep 06 19:38:35.893582 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:35.893567    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2fc63c5-be2e-4253-997b-06ae60ed54be" containerName="maas-api"
Sep 06 19:38:35.893759 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:35.893747    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2047fe90-22c3-493f-b441-3d33a0c40964" containerName="payload-pre-processing"
Sep 06 19:38:35.902573 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:35.902548    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7722-3057c9"
Sep 06 19:38:35.903360 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:35.903334    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7722-3057c9"]
Sep 06 19:38:35.904956 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:35.904930    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zsskm\""
Sep 06 19:38:36.052677 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:36.052644    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n8nq\" (UniqueName: \"kubernetes.io/projected/81fa1d8d-71ff-4c1b-aebb-a670af72989f-kube-api-access-5n8nq\") pod \"mt-curl-7722-3057c9\" (UID: \"81fa1d8d-71ff-4c1b-aebb-a670af72989f\") " pod="opendatahub/mt-curl-7722-3057c9"
Sep 06 19:38:36.153579 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:36.153493    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n8nq\" (UniqueName: \"kubernetes.io/projected/81fa1d8d-71ff-4c1b-aebb-a670af72989f-kube-api-access-5n8nq\") pod \"mt-curl-7722-3057c9\" (UID: \"81fa1d8d-71ff-4c1b-aebb-a670af72989f\") " pod="opendatahub/mt-curl-7722-3057c9"
Sep 06 19:38:36.161615 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:36.161585    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n8nq\" (UniqueName: \"kubernetes.io/projected/81fa1d8d-71ff-4c1b-aebb-a670af72989f-kube-api-access-5n8nq\") pod \"mt-curl-7722-3057c9\" (UID: \"81fa1d8d-71ff-4c1b-aebb-a670af72989f\") " pod="opendatahub/mt-curl-7722-3057c9"
Sep 06 19:38:36.212861 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:36.212841    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7722-3057c9"
Sep 06 19:38:36.483081 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:36.483059    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7722-3057c9"]
Sep 06 19:38:36.485580 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:38:36.485556    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81fa1d8d_71ff_4c1b_aebb_a670af72989f.slice/crio-11c7c4eb54c0da2fdc8979ce5587642f6399fcaf6849e4347a33bbcdae57e014 WatchSource:0}: Error finding container 11c7c4eb54c0da2fdc8979ce5587642f6399fcaf6849e4347a33bbcdae57e014: Status 404 returned error can't find the container with id 11c7c4eb54c0da2fdc8979ce5587642f6399fcaf6849e4347a33bbcdae57e014
Sep 06 19:38:36.486869 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:36.486850    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 19:38:36.632166 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:36.632132    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7722-3057c9" event={"ID":"81fa1d8d-71ff-4c1b-aebb-a670af72989f","Type":"ContainerStarted","Data":"11c7c4eb54c0da2fdc8979ce5587642f6399fcaf6849e4347a33bbcdae57e014"}
Sep 06 19:38:37.639792 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:37.639759    2566 generic.go:356] "Generic (PLEG): container finished" podID="81fa1d8d-71ff-4c1b-aebb-a670af72989f" containerID="4bc8109950316ba7f21b3551d6f890a44d3d109056c6fd6b43264ebec9e95978" exitCode=0
Sep 06 19:38:37.640189 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:37.639843    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7722-3057c9" event={"ID":"81fa1d8d-71ff-4c1b-aebb-a670af72989f","Type":"ContainerDied","Data":"4bc8109950316ba7f21b3551d6f890a44d3d109056c6fd6b43264ebec9e95978"}
Sep 06 19:38:38.781717 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:38.781694    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7722-3057c9"
Sep 06 19:38:38.799397 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:38.799372    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7722-3057c9_81fa1d8d-71ff-4c1b-aebb-a670af72989f/mt-curl-7722-3057c9/0.log"
Sep 06 19:38:38.820344 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:38.820313    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7722-3057c9"]
Sep 06 19:38:38.822773 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:38.822755    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7722-3057c9"]
Sep 06 19:38:38.879197 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:38.879175    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n8nq\" (UniqueName: \"kubernetes.io/projected/81fa1d8d-71ff-4c1b-aebb-a670af72989f-kube-api-access-5n8nq\") pod \"81fa1d8d-71ff-4c1b-aebb-a670af72989f\" (UID: \"81fa1d8d-71ff-4c1b-aebb-a670af72989f\") "
Sep 06 19:38:38.881267 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:38.881244    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fa1d8d-71ff-4c1b-aebb-a670af72989f-kube-api-access-5n8nq" (OuterVolumeSpecName: "kube-api-access-5n8nq") pod "81fa1d8d-71ff-4c1b-aebb-a670af72989f" (UID: "81fa1d8d-71ff-4c1b-aebb-a670af72989f"). InnerVolumeSpecName "kube-api-access-5n8nq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:38:38.932816 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:38.932755    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7722-367721"]
Sep 06 19:38:38.933153 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:38.933141    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81fa1d8d-71ff-4c1b-aebb-a670af72989f" containerName="mt-curl-7722-3057c9"
Sep 06 19:38:38.933190 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:38.933155    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="81fa1d8d-71ff-4c1b-aebb-a670af72989f" containerName="mt-curl-7722-3057c9"
Sep 06 19:38:38.933245 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:38.933237    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="81fa1d8d-71ff-4c1b-aebb-a670af72989f" containerName="mt-curl-7722-3057c9"
Sep 06 19:38:38.936211 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:38.936197    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7722-367721"
Sep 06 19:38:38.942001 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:38.941980    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7722-367721"]
Sep 06 19:38:38.979979 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:38.979952    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n8nq\" (UniqueName: \"kubernetes.io/projected/81fa1d8d-71ff-4c1b-aebb-a670af72989f-kube-api-access-5n8nq\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:38:39.080825 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:39.080799    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-455d4\" (UniqueName: \"kubernetes.io/projected/05496970-7b8e-471f-a02c-fbaa6138a54f-kube-api-access-455d4\") pod \"mt-curl-7722-367721\" (UID: \"05496970-7b8e-471f-a02c-fbaa6138a54f\") " pod="opendatahub/mt-curl-7722-367721"
Sep 06 19:38:39.181521 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:39.181493    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-455d4\" (UniqueName: \"kubernetes.io/projected/05496970-7b8e-471f-a02c-fbaa6138a54f-kube-api-access-455d4\") pod \"mt-curl-7722-367721\" (UID: \"05496970-7b8e-471f-a02c-fbaa6138a54f\") " pod="opendatahub/mt-curl-7722-367721"
Sep 06 19:38:39.189334 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:39.189273    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-455d4\" (UniqueName: \"kubernetes.io/projected/05496970-7b8e-471f-a02c-fbaa6138a54f-kube-api-access-455d4\") pod \"mt-curl-7722-367721\" (UID: \"05496970-7b8e-471f-a02c-fbaa6138a54f\") " pod="opendatahub/mt-curl-7722-367721"
Sep 06 19:38:39.246663 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:39.246639    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7722-367721"
Sep 06 19:38:39.519453 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:39.519426    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7722-367721"]
Sep 06 19:38:39.520518 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:38:39.520493    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05496970_7b8e_471f_a02c_fbaa6138a54f.slice/crio-acb53abb1b54a8b9cbb5f1f71f391ce754146555f29880174855d3c4dfb0fbc7 WatchSource:0}: Error finding container acb53abb1b54a8b9cbb5f1f71f391ce754146555f29880174855d3c4dfb0fbc7: Status 404 returned error can't find the container with id acb53abb1b54a8b9cbb5f1f71f391ce754146555f29880174855d3c4dfb0fbc7
Sep 06 19:38:39.651980 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:39.651946    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7722-367721" event={"ID":"05496970-7b8e-471f-a02c-fbaa6138a54f","Type":"ContainerStarted","Data":"acb53abb1b54a8b9cbb5f1f71f391ce754146555f29880174855d3c4dfb0fbc7"}
Sep 06 19:38:39.653303 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:39.653286    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11c7c4eb54c0da2fdc8979ce5587642f6399fcaf6849e4347a33bbcdae57e014"
Sep 06 19:38:39.653420 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:39.653311    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7722-3057c9"
Sep 06 19:38:39.663335 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:38:39.663310    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7722-3057c9\" is forbidden: User \"system:node:ip-10-0-129-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-252.ec2.internal' and this object" podUID="81fa1d8d-71ff-4c1b-aebb-a670af72989f" pod="opendatahub/mt-curl-7722-3057c9"
Sep 06 19:38:40.310741 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:40.310710    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81fa1d8d-71ff-4c1b-aebb-a670af72989f" path="/var/lib/kubelet/pods/81fa1d8d-71ff-4c1b-aebb-a670af72989f/volumes"
Sep 06 19:38:40.660670 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:40.660551    2566 generic.go:356] "Generic (PLEG): container finished" podID="05496970-7b8e-471f-a02c-fbaa6138a54f" containerID="7a33d05a44e6d204f412a53ffdd795a83f13b1e08571824703f4107521b9d1da" exitCode=0
Sep 06 19:38:40.660670 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:40.660639    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7722-367721" event={"ID":"05496970-7b8e-471f-a02c-fbaa6138a54f","Type":"ContainerDied","Data":"7a33d05a44e6d204f412a53ffdd795a83f13b1e08571824703f4107521b9d1da"}
Sep 06 19:38:41.803837 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:41.803815    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7722-367721"
Sep 06 19:38:41.821749 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:41.821725    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7722-367721_05496970-7b8e-471f-a02c-fbaa6138a54f/mt-curl-7722-367721/0.log"
Sep 06 19:38:41.843413 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:41.843390    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7722-367721"]
Sep 06 19:38:41.846763 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:41.846741    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7722-367721"]
Sep 06 19:38:41.908324 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:41.908302    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-455d4\" (UniqueName: \"kubernetes.io/projected/05496970-7b8e-471f-a02c-fbaa6138a54f-kube-api-access-455d4\") pod \"05496970-7b8e-471f-a02c-fbaa6138a54f\" (UID: \"05496970-7b8e-471f-a02c-fbaa6138a54f\") "
Sep 06 19:38:41.910471 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:41.910445    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05496970-7b8e-471f-a02c-fbaa6138a54f-kube-api-access-455d4" (OuterVolumeSpecName: "kube-api-access-455d4") pod "05496970-7b8e-471f-a02c-fbaa6138a54f" (UID: "05496970-7b8e-471f-a02c-fbaa6138a54f"). InnerVolumeSpecName "kube-api-access-455d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:38:42.009715 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:42.009692    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-455d4\" (UniqueName: \"kubernetes.io/projected/05496970-7b8e-471f-a02c-fbaa6138a54f-kube-api-access-455d4\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:38:42.310991 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:42.310911    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05496970-7b8e-471f-a02c-fbaa6138a54f" path="/var/lib/kubelet/pods/05496970-7b8e-471f-a02c-fbaa6138a54f/volumes"
Sep 06 19:38:42.673236 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:42.673162    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7722-367721"
Sep 06 19:38:42.673236 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:42.673209    2566 scope.go:117] "RemoveContainer" containerID="7a33d05a44e6d204f412a53ffdd795a83f13b1e08571824703f4107521b9d1da"
Sep 06 19:38:42.746433 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:42.746405    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:38:43.779777 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:43.779736    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"]
Sep 06 19:38:46.263351 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.263317    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"]
Sep 06 19:38:46.263769 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.263564    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd" podUID="48c4f9d2-3e07-4f6c-af7a-7c796a894bf8" containerName="maas-api" containerID="cri-o://ac3c853e782f633e3ba25dd05cb9f94b4e7d4a49b808905afe0dc16ed99df184" gracePeriod=30
Sep 06 19:38:46.318887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.318856    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp"]
Sep 06 19:38:46.319114 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.319063    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp" podUID="ae2de6c6-8885-486d-bac3-92bef9a3bfc0" containerName="payload-pre-processing" containerID="cri-o://9eafa43d97083fdd0ce6d425cad6024ee0390cebcd3a7af4e51b4135dfb80035" gracePeriod=30
Sep 06 19:38:46.562666 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.562642    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"
Sep 06 19:38:46.597847 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.597823    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp"
Sep 06 19:38:46.697229 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.697194    2566 generic.go:356] "Generic (PLEG): container finished" podID="ae2de6c6-8885-486d-bac3-92bef9a3bfc0" containerID="9eafa43d97083fdd0ce6d425cad6024ee0390cebcd3a7af4e51b4135dfb80035" exitCode=0
Sep 06 19:38:46.697422 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.697242    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp"
Sep 06 19:38:46.697422 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.697277    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp" event={"ID":"ae2de6c6-8885-486d-bac3-92bef9a3bfc0","Type":"ContainerDied","Data":"9eafa43d97083fdd0ce6d425cad6024ee0390cebcd3a7af4e51b4135dfb80035"}
Sep 06 19:38:46.697422 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.697331    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp" event={"ID":"ae2de6c6-8885-486d-bac3-92bef9a3bfc0","Type":"ContainerDied","Data":"1198895e37ce43b2abb2e44cf83383b0bf9544f82f3be836f6ca2127677f0ae9"}
Sep 06 19:38:46.697422 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.697364    2566 scope.go:117] "RemoveContainer" containerID="9eafa43d97083fdd0ce6d425cad6024ee0390cebcd3a7af4e51b4135dfb80035"
Sep 06 19:38:46.698651 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.698633    2566 generic.go:356] "Generic (PLEG): container finished" podID="48c4f9d2-3e07-4f6c-af7a-7c796a894bf8" containerID="ac3c853e782f633e3ba25dd05cb9f94b4e7d4a49b808905afe0dc16ed99df184" exitCode=0
Sep 06 19:38:46.698735 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.698697    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"
Sep 06 19:38:46.698735 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.698720    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd" event={"ID":"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8","Type":"ContainerDied","Data":"ac3c853e782f633e3ba25dd05cb9f94b4e7d4a49b808905afe0dc16ed99df184"}
Sep 06 19:38:46.698833 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.698759    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd" event={"ID":"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8","Type":"ContainerDied","Data":"19af85d08035fe799da70bff09bc6edf05773bb7a4814b3f5874d475b4c08d58"}
Sep 06 19:38:46.706500 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.706480    2566 scope.go:117] "RemoveContainer" containerID="9eafa43d97083fdd0ce6d425cad6024ee0390cebcd3a7af4e51b4135dfb80035"
Sep 06 19:38:46.706789 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:38:46.706771    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eafa43d97083fdd0ce6d425cad6024ee0390cebcd3a7af4e51b4135dfb80035\": container with ID starting with 9eafa43d97083fdd0ce6d425cad6024ee0390cebcd3a7af4e51b4135dfb80035 not found: ID does not exist" containerID="9eafa43d97083fdd0ce6d425cad6024ee0390cebcd3a7af4e51b4135dfb80035"
Sep 06 19:38:46.706834 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.706798    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eafa43d97083fdd0ce6d425cad6024ee0390cebcd3a7af4e51b4135dfb80035"} err="failed to get container status \"9eafa43d97083fdd0ce6d425cad6024ee0390cebcd3a7af4e51b4135dfb80035\": rpc error: code = NotFound desc = could not find container \"9eafa43d97083fdd0ce6d425cad6024ee0390cebcd3a7af4e51b4135dfb80035\": container with ID starting with 9eafa43d97083fdd0ce6d425cad6024ee0390cebcd3a7af4e51b4135dfb80035 not found: ID does not exist"
Sep 06 19:38:46.706834 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.706815    2566 scope.go:117] "RemoveContainer" containerID="ac3c853e782f633e3ba25dd05cb9f94b4e7d4a49b808905afe0dc16ed99df184"
Sep 06 19:38:46.715215 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.715198    2566 scope.go:117] "RemoveContainer" containerID="ac3c853e782f633e3ba25dd05cb9f94b4e7d4a49b808905afe0dc16ed99df184"
Sep 06 19:38:46.715467 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:38:46.715449    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3c853e782f633e3ba25dd05cb9f94b4e7d4a49b808905afe0dc16ed99df184\": container with ID starting with ac3c853e782f633e3ba25dd05cb9f94b4e7d4a49b808905afe0dc16ed99df184 not found: ID does not exist" containerID="ac3c853e782f633e3ba25dd05cb9f94b4e7d4a49b808905afe0dc16ed99df184"
Sep 06 19:38:46.715513 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.715474    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3c853e782f633e3ba25dd05cb9f94b4e7d4a49b808905afe0dc16ed99df184"} err="failed to get container status \"ac3c853e782f633e3ba25dd05cb9f94b4e7d4a49b808905afe0dc16ed99df184\": rpc error: code = NotFound desc = could not find container \"ac3c853e782f633e3ba25dd05cb9f94b4e7d4a49b808905afe0dc16ed99df184\": container with ID starting with ac3c853e782f633e3ba25dd05cb9f94b4e7d4a49b808905afe0dc16ed99df184 not found: ID does not exist"
Sep 06 19:38:46.751813 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.751796    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zv4sw\" (UniqueName: \"kubernetes.io/projected/ae2de6c6-8885-486d-bac3-92bef9a3bfc0-kube-api-access-zv4sw\") pod \"ae2de6c6-8885-486d-bac3-92bef9a3bfc0\" (UID: \"ae2de6c6-8885-486d-bac3-92bef9a3bfc0\") "
Sep 06 19:38:46.751927 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.751879    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae2de6c6-8885-486d-bac3-92bef9a3bfc0-plugins-config-volume\") pod \"ae2de6c6-8885-486d-bac3-92bef9a3bfc0\" (UID: \"ae2de6c6-8885-486d-bac3-92bef9a3bfc0\") "
Sep 06 19:38:46.751979 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.751939    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-openshift-service-ca\") pod \"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8\" (UID: \"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8\") "
Sep 06 19:38:46.751979 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.751960    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9k4x\" (UniqueName: \"kubernetes.io/projected/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-kube-api-access-w9k4x\") pod \"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8\" (UID: \"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8\") "
Sep 06 19:38:46.752075 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.752007    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-metrics-certs\") pod \"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8\" (UID: \"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8\") "
Sep 06 19:38:46.752127 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.752075    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-maas-api-tls\") pod \"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8\" (UID: \"48c4f9d2-3e07-4f6c-af7a-7c796a894bf8\") "
Sep 06 19:38:46.752299 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.752268    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "48c4f9d2-3e07-4f6c-af7a-7c796a894bf8" (UID: "48c4f9d2-3e07-4f6c-af7a-7c796a894bf8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:38:46.752403 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.752387    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-openshift-service-ca\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:38:46.753257 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.753225    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2de6c6-8885-486d-bac3-92bef9a3bfc0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ae2de6c6-8885-486d-bac3-92bef9a3bfc0" (UID: "ae2de6c6-8885-486d-bac3-92bef9a3bfc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:38:46.754578 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.754526    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "48c4f9d2-3e07-4f6c-af7a-7c796a894bf8" (UID: "48c4f9d2-3e07-4f6c-af7a-7c796a894bf8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:38:46.754773 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.754743    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-kube-api-access-w9k4x" (OuterVolumeSpecName: "kube-api-access-w9k4x") pod "48c4f9d2-3e07-4f6c-af7a-7c796a894bf8" (UID: "48c4f9d2-3e07-4f6c-af7a-7c796a894bf8"). InnerVolumeSpecName "kube-api-access-w9k4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:38:46.754871 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.754750    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "48c4f9d2-3e07-4f6c-af7a-7c796a894bf8" (UID: "48c4f9d2-3e07-4f6c-af7a-7c796a894bf8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:38:46.754871 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.754802    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2de6c6-8885-486d-bac3-92bef9a3bfc0-kube-api-access-zv4sw" (OuterVolumeSpecName: "kube-api-access-zv4sw") pod "ae2de6c6-8885-486d-bac3-92bef9a3bfc0" (UID: "ae2de6c6-8885-486d-bac3-92bef9a3bfc0"). InnerVolumeSpecName "kube-api-access-zv4sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:38:46.853399 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.853305    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-maas-api-tls\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:38:46.853399 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.853341    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv4sw\" (UniqueName: \"kubernetes.io/projected/ae2de6c6-8885-486d-bac3-92bef9a3bfc0-kube-api-access-zv4sw\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:38:46.853399 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.853382    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae2de6c6-8885-486d-bac3-92bef9a3bfc0-plugins-config-volume\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:38:46.853399 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.853397    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9k4x\" (UniqueName: \"kubernetes.io/projected/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-kube-api-access-w9k4x\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:38:46.853678 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:46.853411    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8-metrics-certs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:38:47.032402 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:47.032375    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"]
Sep 06 19:38:47.036049 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:47.036024    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49a2d2-6bc5c7cb7f-r68jd"]
Sep 06 19:38:47.053391 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:47.053370    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp"]
Sep 06 19:38:47.061020 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:47.061000    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49a2d2-d7c7b6c5d-pgrzp"]
Sep 06 19:38:48.311020 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:48.310984    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48c4f9d2-3e07-4f6c-af7a-7c796a894bf8" path="/var/lib/kubelet/pods/48c4f9d2-3e07-4f6c-af7a-7c796a894bf8/volumes"
Sep 06 19:38:48.311499 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:48.311478    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae2de6c6-8885-486d-bac3-92bef9a3bfc0" path="/var/lib/kubelet/pods/ae2de6c6-8885-486d-bac3-92bef9a3bfc0/volumes"
Sep 06 19:38:51.539833 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.539800    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"]
Sep 06 19:38:51.540259 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.540244    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05496970-7b8e-471f-a02c-fbaa6138a54f" containerName="mt-curl-7722-367721"
Sep 06 19:38:51.540305 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.540262    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="05496970-7b8e-471f-a02c-fbaa6138a54f" containerName="mt-curl-7722-367721"
Sep 06 19:38:51.540305 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.540279    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48c4f9d2-3e07-4f6c-af7a-7c796a894bf8" containerName="maas-api"
Sep 06 19:38:51.540305 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.540286    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="48c4f9d2-3e07-4f6c-af7a-7c796a894bf8" containerName="maas-api"
Sep 06 19:38:51.540305 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.540293    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae2de6c6-8885-486d-bac3-92bef9a3bfc0" containerName="payload-pre-processing"
Sep 06 19:38:51.540305 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.540298    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae2de6c6-8885-486d-bac3-92bef9a3bfc0" containerName="payload-pre-processing"
Sep 06 19:38:51.540498 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.540365    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="48c4f9d2-3e07-4f6c-af7a-7c796a894bf8" containerName="maas-api"
Sep 06 19:38:51.540498 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.540373    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae2de6c6-8885-486d-bac3-92bef9a3bfc0" containerName="payload-pre-processing"
Sep 06 19:38:51.540498 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.540381    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="05496970-7b8e-471f-a02c-fbaa6138a54f" containerName="mt-curl-7722-367721"
Sep 06 19:38:51.544927 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.544908    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:38:51.547712 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.547678    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e5ef25\"/\"rate-test-model-e5ef25-kserve-self-signed-certs\""
Sep 06 19:38:51.549968 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.549944    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"]
Sep 06 19:38:51.594006 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.593972    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-home\") pod \"rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:38:51.594145 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.594020    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-tmp-dir\") pod \"rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:38:51.594145 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.594118    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-model-cache\") pod \"rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:38:51.594260 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.594154    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-dshm\") pod \"rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:38:51.594260 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.594186    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52rj9\" (UniqueName: \"kubernetes.io/projected/c73b4621-5eb3-4893-afe3-673cc1cbfd53-kube-api-access-52rj9\") pod \"rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:38:51.594260 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.594212    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c73b4621-5eb3-4893-afe3-673cc1cbfd53-tls-certs\") pod \"rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:38:51.694857 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.694824    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-home\") pod \"rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:38:51.694995 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.694865    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-tmp-dir\") pod \"rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:38:51.694995 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.694947    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-model-cache\") pod \"rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:38:51.694995 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.694985    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-dshm\") pod \"rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:38:51.695168 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.695014    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52rj9\" (UniqueName: \"kubernetes.io/projected/c73b4621-5eb3-4893-afe3-673cc1cbfd53-kube-api-access-52rj9\") pod \"rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:38:51.695168 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.695040    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c73b4621-5eb3-4893-afe3-673cc1cbfd53-tls-certs\") pod \"rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:38:51.695269 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.695256    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-home\") pod \"rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:38:51.695331 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.695311    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-tmp-dir\") pod \"rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:38:51.695412 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.695323    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-model-cache\") pod \"rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:38:51.697381 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.697354    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-dshm\") pod \"rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:38:51.697636 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.697620    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c73b4621-5eb3-4893-afe3-673cc1cbfd53-tls-certs\") pod \"rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:38:51.702665 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.702642    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52rj9\" (UniqueName: \"kubernetes.io/projected/c73b4621-5eb3-4893-afe3-673cc1cbfd53-kube-api-access-52rj9\") pod \"rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") " pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:38:51.855346 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:51.855260    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:38:52.129802 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:52.129776    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"]
Sep 06 19:38:52.131513 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:38:52.131485    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc73b4621_5eb3_4893_afe3_673cc1cbfd53.slice/crio-653bc86f82a69a30753a291725717040cce01c80f92e892a646fa788c40aeb6b WatchSource:0}: Error finding container 653bc86f82a69a30753a291725717040cce01c80f92e892a646fa788c40aeb6b: Status 404 returned error can't find the container with id 653bc86f82a69a30753a291725717040cce01c80f92e892a646fa788c40aeb6b
Sep 06 19:38:52.736863 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:52.736829    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs" event={"ID":"c73b4621-5eb3-4893-afe3-673cc1cbfd53","Type":"ContainerStarted","Data":"f3a62ff0efd7bb23aa2cb737f1b3e24872a28ad9122fd68becf0238b858d501c"}
Sep 06 19:38:52.737316 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:52.736870    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs" event={"ID":"c73b4621-5eb3-4893-afe3-673cc1cbfd53","Type":"ContainerStarted","Data":"653bc86f82a69a30753a291725717040cce01c80f92e892a646fa788c40aeb6b"}
Sep 06 19:38:52.753421 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:52.753377    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs" podStartSLOduration=1.753362615 podStartE2EDuration="1.753362615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:38:52.751579307 +0000 UTC m=+1666.932761115" watchObservedRunningTime="2026-09-06 19:38:52.753362615 +0000 UTC m=+1666.934544421"
Sep 06 19:38:57.847126 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:57.847096    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:38:58.799531 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:58.799491    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd" podUID="bb9fed6a-8027-4e3d-9305-a96426d7c0aa" containerName="main" containerID="cri-o://c7973a6b357f3f6a5c02afd9ee9e8839b1b4b14e697d4dc9d97ed4499e8a57ad" gracePeriod=45
Sep 06 19:38:59.038793 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.038770    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:38:59.168043 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.167971    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lvnw\" (UniqueName: \"kubernetes.io/projected/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-kube-api-access-4lvnw\") pod \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") "
Sep 06 19:38:59.168197 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.168053    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-model-cache\") pod \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") "
Sep 06 19:38:59.168197 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.168098    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-tmp-dir\") pod \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") "
Sep 06 19:38:59.168197 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.168132    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-dshm\") pod \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") "
Sep 06 19:38:59.168197 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.168159    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-tls-certs\") pod \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") "
Sep 06 19:38:59.168409 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.168225    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-home\") pod \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\" (UID: \"bb9fed6a-8027-4e3d-9305-a96426d7c0aa\") "
Sep 06 19:38:59.168409 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.168333    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb9fed6a-8027-4e3d-9305-a96426d7c0aa" (UID: "bb9fed6a-8027-4e3d-9305-a96426d7c0aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:38:59.168409 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.168395    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb9fed6a-8027-4e3d-9305-a96426d7c0aa" (UID: "bb9fed6a-8027-4e3d-9305-a96426d7c0aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:38:59.168562 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.168506    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-home" (OuterVolumeSpecName: "home") pod "bb9fed6a-8027-4e3d-9305-a96426d7c0aa" (UID: "bb9fed6a-8027-4e3d-9305-a96426d7c0aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:38:59.168562 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.168550    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-model-cache\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:38:59.168690 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.168568    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-tmp-dir\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:38:59.170255 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.170232    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-kube-api-access-4lvnw" (OuterVolumeSpecName: "kube-api-access-4lvnw") pod "bb9fed6a-8027-4e3d-9305-a96426d7c0aa" (UID: "bb9fed6a-8027-4e3d-9305-a96426d7c0aa"). InnerVolumeSpecName "kube-api-access-4lvnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:38:59.170376 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.170329    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb9fed6a-8027-4e3d-9305-a96426d7c0aa" (UID: "bb9fed6a-8027-4e3d-9305-a96426d7c0aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:38:59.170376 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.170343    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-dshm" (OuterVolumeSpecName: "dshm") pod "bb9fed6a-8027-4e3d-9305-a96426d7c0aa" (UID: "bb9fed6a-8027-4e3d-9305-a96426d7c0aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:38:59.269312 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.269290    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-home\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:38:59.269312 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.269312    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lvnw\" (UniqueName: \"kubernetes.io/projected/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-kube-api-access-4lvnw\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:38:59.269464 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.269322    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-dshm\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:38:59.269464 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.269331    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9fed6a-8027-4e3d-9305-a96426d7c0aa-tls-certs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:38:59.777767 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.777734    2566 generic.go:356] "Generic (PLEG): container finished" podID="bb9fed6a-8027-4e3d-9305-a96426d7c0aa" containerID="c7973a6b357f3f6a5c02afd9ee9e8839b1b4b14e697d4dc9d97ed4499e8a57ad" exitCode=0
Sep 06 19:38:59.777945 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.777780    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"
Sep 06 19:38:59.777945 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.777812    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd" event={"ID":"bb9fed6a-8027-4e3d-9305-a96426d7c0aa","Type":"ContainerDied","Data":"c7973a6b357f3f6a5c02afd9ee9e8839b1b4b14e697d4dc9d97ed4499e8a57ad"}
Sep 06 19:38:59.777945 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.777851    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd" event={"ID":"bb9fed6a-8027-4e3d-9305-a96426d7c0aa","Type":"ContainerDied","Data":"27f09df507ed153f2819b86dc169bd73e2445ec75f9ab915dfd88551542d3150"}
Sep 06 19:38:59.778097 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.777952    2566 scope.go:117] "RemoveContainer" containerID="c7973a6b357f3f6a5c02afd9ee9e8839b1b4b14e697d4dc9d97ed4499e8a57ad"
Sep 06 19:38:59.787814 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.787800    2566 scope.go:117] "RemoveContainer" containerID="c7973a6b357f3f6a5c02afd9ee9e8839b1b4b14e697d4dc9d97ed4499e8a57ad"
Sep 06 19:38:59.788117 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:38:59.788096    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7973a6b357f3f6a5c02afd9ee9e8839b1b4b14e697d4dc9d97ed4499e8a57ad\": container with ID starting with c7973a6b357f3f6a5c02afd9ee9e8839b1b4b14e697d4dc9d97ed4499e8a57ad not found: ID does not exist" containerID="c7973a6b357f3f6a5c02afd9ee9e8839b1b4b14e697d4dc9d97ed4499e8a57ad"
Sep 06 19:38:59.788187 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.788128    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7973a6b357f3f6a5c02afd9ee9e8839b1b4b14e697d4dc9d97ed4499e8a57ad"} err="failed to get container status \"c7973a6b357f3f6a5c02afd9ee9e8839b1b4b14e697d4dc9d97ed4499e8a57ad\": rpc error: code = NotFound desc = could not find container \"c7973a6b357f3f6a5c02afd9ee9e8839b1b4b14e697d4dc9d97ed4499e8a57ad\": container with ID starting with c7973a6b357f3f6a5c02afd9ee9e8839b1b4b14e697d4dc9d97ed4499e8a57ad not found: ID does not exist"
Sep 06 19:38:59.798358 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.798334    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"]
Sep 06 19:38:59.803294 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:38:59.803272    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5ef25/sub-test-model-e5ef25-kserve-649cf654c-55vfd"]
Sep 06 19:39:00.310496 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:00.310464    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb9fed6a-8027-4e3d-9305-a96426d7c0aa" path="/var/lib/kubelet/pods/bb9fed6a-8027-4e3d-9305-a96426d7c0aa/volumes"
Sep 06 19:39:01.855595 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:01.855563    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:39:01.855989 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:01.855797    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:39:01.868126 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:01.868100    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:39:02.807410 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:02.807381    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:39:08.451544 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:08.451513    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:39:20.847292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:20.847257    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"]
Sep 06 19:39:20.847823 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:20.847501    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr" podUID="f54482c5-1e86-4580-95d5-6e1504bbc177" containerName="maas-api" containerID="cri-o://3b2e0fb682386203956b8fd92654b5c05c7bc990fd8817546d26c9c6d717a774" gracePeriod=30
Sep 06 19:39:21.094415 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:21.094389    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"
Sep 06 19:39:21.271282 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:21.271244    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5ggs\" (UniqueName: \"kubernetes.io/projected/f54482c5-1e86-4580-95d5-6e1504bbc177-kube-api-access-c5ggs\") pod \"f54482c5-1e86-4580-95d5-6e1504bbc177\" (UID: \"f54482c5-1e86-4580-95d5-6e1504bbc177\") "
Sep 06 19:39:21.271282 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:21.271284    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f54482c5-1e86-4580-95d5-6e1504bbc177-openshift-service-ca\") pod \"f54482c5-1e86-4580-95d5-6e1504bbc177\" (UID: \"f54482c5-1e86-4580-95d5-6e1504bbc177\") "
Sep 06 19:39:21.271542 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:21.271303    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f54482c5-1e86-4580-95d5-6e1504bbc177-maas-api-tls\") pod \"f54482c5-1e86-4580-95d5-6e1504bbc177\" (UID: \"f54482c5-1e86-4580-95d5-6e1504bbc177\") "
Sep 06 19:39:21.271542 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:21.271357    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f54482c5-1e86-4580-95d5-6e1504bbc177-metrics-certs\") pod \"f54482c5-1e86-4580-95d5-6e1504bbc177\" (UID: \"f54482c5-1e86-4580-95d5-6e1504bbc177\") "
Sep 06 19:39:21.271755 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:21.271707    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54482c5-1e86-4580-95d5-6e1504bbc177-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f54482c5-1e86-4580-95d5-6e1504bbc177" (UID: "f54482c5-1e86-4580-95d5-6e1504bbc177"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:39:21.273628 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:21.273571    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54482c5-1e86-4580-95d5-6e1504bbc177-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f54482c5-1e86-4580-95d5-6e1504bbc177" (UID: "f54482c5-1e86-4580-95d5-6e1504bbc177"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:39:21.273706 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:21.273628    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54482c5-1e86-4580-95d5-6e1504bbc177-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f54482c5-1e86-4580-95d5-6e1504bbc177" (UID: "f54482c5-1e86-4580-95d5-6e1504bbc177"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:39:21.273706 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:21.273641    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54482c5-1e86-4580-95d5-6e1504bbc177-kube-api-access-c5ggs" (OuterVolumeSpecName: "kube-api-access-c5ggs") pod "f54482c5-1e86-4580-95d5-6e1504bbc177" (UID: "f54482c5-1e86-4580-95d5-6e1504bbc177"). InnerVolumeSpecName "kube-api-access-c5ggs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:39:21.372489 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:21.372454    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f54482c5-1e86-4580-95d5-6e1504bbc177-metrics-certs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:39:21.372489 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:21.372485    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5ggs\" (UniqueName: \"kubernetes.io/projected/f54482c5-1e86-4580-95d5-6e1504bbc177-kube-api-access-c5ggs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:39:21.372697 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:21.372500    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f54482c5-1e86-4580-95d5-6e1504bbc177-openshift-service-ca\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:39:21.372697 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:21.372511    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f54482c5-1e86-4580-95d5-6e1504bbc177-maas-api-tls\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:39:21.909685 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:21.909650    2566 generic.go:356] "Generic (PLEG): container finished" podID="f54482c5-1e86-4580-95d5-6e1504bbc177" containerID="3b2e0fb682386203956b8fd92654b5c05c7bc990fd8817546d26c9c6d717a774" exitCode=0
Sep 06 19:39:21.910105 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:21.909712    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"
Sep 06 19:39:21.910105 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:21.909742    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr" event={"ID":"f54482c5-1e86-4580-95d5-6e1504bbc177","Type":"ContainerDied","Data":"3b2e0fb682386203956b8fd92654b5c05c7bc990fd8817546d26c9c6d717a774"}
Sep 06 19:39:21.910105 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:21.909822    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr" event={"ID":"f54482c5-1e86-4580-95d5-6e1504bbc177","Type":"ContainerDied","Data":"cd29310cc9151dce5a885bee46339fb15a6fe588276154592a258ce82a5889fd"}
Sep 06 19:39:21.910105 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:21.909835    2566 scope.go:117] "RemoveContainer" containerID="3b2e0fb682386203956b8fd92654b5c05c7bc990fd8817546d26c9c6d717a774"
Sep 06 19:39:21.920010 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:21.919996    2566 scope.go:117] "RemoveContainer" containerID="3b2e0fb682386203956b8fd92654b5c05c7bc990fd8817546d26c9c6d717a774"
Sep 06 19:39:21.920264 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:39:21.920247    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b2e0fb682386203956b8fd92654b5c05c7bc990fd8817546d26c9c6d717a774\": container with ID starting with 3b2e0fb682386203956b8fd92654b5c05c7bc990fd8817546d26c9c6d717a774 not found: ID does not exist" containerID="3b2e0fb682386203956b8fd92654b5c05c7bc990fd8817546d26c9c6d717a774"
Sep 06 19:39:21.920321 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:21.920274    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2e0fb682386203956b8fd92654b5c05c7bc990fd8817546d26c9c6d717a774"} err="failed to get container status \"3b2e0fb682386203956b8fd92654b5c05c7bc990fd8817546d26c9c6d717a774\": rpc error: code = NotFound desc = could not find container \"3b2e0fb682386203956b8fd92654b5c05c7bc990fd8817546d26c9c6d717a774\": container with ID starting with 3b2e0fb682386203956b8fd92654b5c05c7bc990fd8817546d26c9c6d717a774 not found: ID does not exist"
Sep 06 19:39:21.931625 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:21.931591    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"]
Sep 06 19:39:21.933827 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:21.933809    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6bbb53-56795dd4c4-jn5pr"]
Sep 06 19:39:22.310350 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:22.310307    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f54482c5-1e86-4580-95d5-6e1504bbc177" path="/var/lib/kubelet/pods/f54482c5-1e86-4580-95d5-6e1504bbc177/volumes"
Sep 06 19:39:24.449681 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:24.449654    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:39:26.746723 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.746690    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"]
Sep 06 19:39:26.747301 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.747282    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f54482c5-1e86-4580-95d5-6e1504bbc177" containerName="maas-api"
Sep 06 19:39:26.747345 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.747306    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f54482c5-1e86-4580-95d5-6e1504bbc177" containerName="maas-api"
Sep 06 19:39:26.747345 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.747338    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb9fed6a-8027-4e3d-9305-a96426d7c0aa" containerName="main"
Sep 06 19:39:26.747414 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.747347    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb9fed6a-8027-4e3d-9305-a96426d7c0aa" containerName="main"
Sep 06 19:39:26.747466 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.747453    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f54482c5-1e86-4580-95d5-6e1504bbc177" containerName="maas-api"
Sep 06 19:39:26.747502 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.747470    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb9fed6a-8027-4e3d-9305-a96426d7c0aa" containerName="main"
Sep 06 19:39:26.752329 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.752309    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:26.755685 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.755592    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-476f98\"/\"rate-test-model-476f98-kserve-self-signed-certs\""
Sep 06 19:39:26.755950 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.755927    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-476f98\"/\"openshift-service-ca.crt\""
Sep 06 19:39:26.756439 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.756416    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-476f98\"/\"kube-root-ca.crt\""
Sep 06 19:39:26.762107 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.760086    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-476f98\"/\"default-dockercfg-lf6cj\""
Sep 06 19:39:26.764166 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.764135    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"]
Sep 06 19:39:26.821696 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.821668    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-home\") pod \"rate-test-model-476f98-kserve-bd658d75-t2vsp\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") " pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:26.821804 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.821715    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-tmp-dir\") pod \"rate-test-model-476f98-kserve-bd658d75-t2vsp\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") " pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:26.821804 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.821774    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7530715d-3155-4ab0-8e0c-4c9785ee470c-tls-certs\") pod \"rate-test-model-476f98-kserve-bd658d75-t2vsp\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") " pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:26.821804 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.821799    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-model-cache\") pod \"rate-test-model-476f98-kserve-bd658d75-t2vsp\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") " pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:26.821908 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.821844    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgmjl\" (UniqueName: \"kubernetes.io/projected/7530715d-3155-4ab0-8e0c-4c9785ee470c-kube-api-access-bgmjl\") pod \"rate-test-model-476f98-kserve-bd658d75-t2vsp\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") " pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:26.821908 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.821867    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-dshm\") pod \"rate-test-model-476f98-kserve-bd658d75-t2vsp\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") " pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:26.922280 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.922254    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgmjl\" (UniqueName: \"kubernetes.io/projected/7530715d-3155-4ab0-8e0c-4c9785ee470c-kube-api-access-bgmjl\") pod \"rate-test-model-476f98-kserve-bd658d75-t2vsp\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") " pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:26.922390 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.922285    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-dshm\") pod \"rate-test-model-476f98-kserve-bd658d75-t2vsp\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") " pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:26.922390 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.922314    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-home\") pod \"rate-test-model-476f98-kserve-bd658d75-t2vsp\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") " pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:26.922390 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.922355    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-tmp-dir\") pod \"rate-test-model-476f98-kserve-bd658d75-t2vsp\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") " pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:26.922507 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.922393    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7530715d-3155-4ab0-8e0c-4c9785ee470c-tls-certs\") pod \"rate-test-model-476f98-kserve-bd658d75-t2vsp\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") " pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:26.922507 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.922426    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-model-cache\") pod \"rate-test-model-476f98-kserve-bd658d75-t2vsp\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") " pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:26.922829 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.922795    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-home\") pod \"rate-test-model-476f98-kserve-bd658d75-t2vsp\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") " pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:26.922926 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.922809    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-tmp-dir\") pod \"rate-test-model-476f98-kserve-bd658d75-t2vsp\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") " pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:26.922926 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.922859    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-model-cache\") pod \"rate-test-model-476f98-kserve-bd658d75-t2vsp\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") " pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:26.924779 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.924756    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-dshm\") pod \"rate-test-model-476f98-kserve-bd658d75-t2vsp\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") " pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:26.924922 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.924905    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7530715d-3155-4ab0-8e0c-4c9785ee470c-tls-certs\") pod \"rate-test-model-476f98-kserve-bd658d75-t2vsp\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") " pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:26.931053 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:26.931032    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgmjl\" (UniqueName: \"kubernetes.io/projected/7530715d-3155-4ab0-8e0c-4c9785ee470c-kube-api-access-bgmjl\") pod \"rate-test-model-476f98-kserve-bd658d75-t2vsp\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") " pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:27.071995 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:27.071916    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:27.345643 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:27.345584    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"]
Sep 06 19:39:27.948202 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:27.948168    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp" event={"ID":"7530715d-3155-4ab0-8e0c-4c9785ee470c","Type":"ContainerStarted","Data":"00274852096eeeb5dd10cf358b71b23682760ef74015859697240378ee239334"}
Sep 06 19:39:27.948541 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:27.948210    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp" event={"ID":"7530715d-3155-4ab0-8e0c-4c9785ee470c","Type":"ContainerStarted","Data":"3f6f4f50bead0afa42e5f9cdd71b167e67efe7675f274655e380472b487d8955"}
Sep 06 19:39:27.966015 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:27.965969    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp" podStartSLOduration=1.965954301 podStartE2EDuration="1.965954301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:39:27.964999041 +0000 UTC m=+1702.146180858" watchObservedRunningTime="2026-09-06 19:39:27.965954301 +0000 UTC m=+1702.147136107"
Sep 06 19:39:36.350094 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:36.350063    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:39:37.072078 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:37.072045    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:37.072251 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:37.072093    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:37.084596 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:37.084572    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:38.015773 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:38.015746    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:39:46.850217 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:39:46.850182    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:40:01.044234 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:01.044198    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:40:06.644282 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:06.644247    2566 scope.go:117] "RemoveContainer" containerID="084211757c40446dc132009a6bd6c70f2e5abd855fbdeee25e572aff7a8255fd"
Sep 06 19:40:06.653397 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:06.653378    2566 scope.go:117] "RemoveContainer" containerID="57006835831fb5221b3380d479bf0bc2aa8903a12290b71fdd763a8c076f2093"
Sep 06 19:40:06.976254 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:06.976223    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"]
Sep 06 19:40:07.951020 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:07.950990    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"]
Sep 06 19:40:11.552464 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:11.552424    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:40:21.450768 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:21.450728    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:40:21.996655 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:21.996582    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs" podUID="c73b4621-5eb3-4893-afe3-673cc1cbfd53" containerName="main" containerID="cri-o://f3a62ff0efd7bb23aa2cb737f1b3e24872a28ad9122fd68becf0238b858d501c" gracePeriod=45
Sep 06 19:40:22.244394 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.244371    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:40:22.256957 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.256907    2566 generic.go:356] "Generic (PLEG): container finished" podID="c73b4621-5eb3-4893-afe3-673cc1cbfd53" containerID="f3a62ff0efd7bb23aa2cb737f1b3e24872a28ad9122fd68becf0238b858d501c" exitCode=0
Sep 06 19:40:22.256957 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.256944    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"
Sep 06 19:40:22.257081 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.256993    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs" event={"ID":"c73b4621-5eb3-4893-afe3-673cc1cbfd53","Type":"ContainerDied","Data":"f3a62ff0efd7bb23aa2cb737f1b3e24872a28ad9122fd68becf0238b858d501c"}
Sep 06 19:40:22.257081 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.257032    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs" event={"ID":"c73b4621-5eb3-4893-afe3-673cc1cbfd53","Type":"ContainerDied","Data":"653bc86f82a69a30753a291725717040cce01c80f92e892a646fa788c40aeb6b"}
Sep 06 19:40:22.257081 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.257057    2566 scope.go:117] "RemoveContainer" containerID="f3a62ff0efd7bb23aa2cb737f1b3e24872a28ad9122fd68becf0238b858d501c"
Sep 06 19:40:22.266997 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.266975    2566 scope.go:117] "RemoveContainer" containerID="f3a62ff0efd7bb23aa2cb737f1b3e24872a28ad9122fd68becf0238b858d501c"
Sep 06 19:40:22.267337 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:40:22.267314    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a62ff0efd7bb23aa2cb737f1b3e24872a28ad9122fd68becf0238b858d501c\": container with ID starting with f3a62ff0efd7bb23aa2cb737f1b3e24872a28ad9122fd68becf0238b858d501c not found: ID does not exist" containerID="f3a62ff0efd7bb23aa2cb737f1b3e24872a28ad9122fd68becf0238b858d501c"
Sep 06 19:40:22.267438 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.267345    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a62ff0efd7bb23aa2cb737f1b3e24872a28ad9122fd68becf0238b858d501c"} err="failed to get container status \"f3a62ff0efd7bb23aa2cb737f1b3e24872a28ad9122fd68becf0238b858d501c\": rpc error: code = NotFound desc = could not find container \"f3a62ff0efd7bb23aa2cb737f1b3e24872a28ad9122fd68becf0238b858d501c\": container with ID starting with f3a62ff0efd7bb23aa2cb737f1b3e24872a28ad9122fd68becf0238b858d501c not found: ID does not exist"
Sep 06 19:40:22.416960 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.416923    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c73b4621-5eb3-4893-afe3-673cc1cbfd53-tls-certs\") pod \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") "
Sep 06 19:40:22.417125 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.416969    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-dshm\") pod \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") "
Sep 06 19:40:22.417125 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.417021    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-model-cache\") pod \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") "
Sep 06 19:40:22.417125 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.417056    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-tmp-dir\") pod \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") "
Sep 06 19:40:22.417125 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.417109    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52rj9\" (UniqueName: \"kubernetes.io/projected/c73b4621-5eb3-4893-afe3-673cc1cbfd53-kube-api-access-52rj9\") pod \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") "
Sep 06 19:40:22.417348 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.417158    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-home\") pod \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\" (UID: \"c73b4621-5eb3-4893-afe3-673cc1cbfd53\") "
Sep 06 19:40:22.417348 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.417313    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-model-cache" (OuterVolumeSpecName: "model-cache") pod "c73b4621-5eb3-4893-afe3-673cc1cbfd53" (UID: "c73b4621-5eb3-4893-afe3-673cc1cbfd53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:40:22.417456 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.417343    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c73b4621-5eb3-4893-afe3-673cc1cbfd53" (UID: "c73b4621-5eb3-4893-afe3-673cc1cbfd53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:40:22.417514 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.417496    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-home" (OuterVolumeSpecName: "home") pod "c73b4621-5eb3-4893-afe3-673cc1cbfd53" (UID: "c73b4621-5eb3-4893-afe3-673cc1cbfd53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:40:22.417580 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.417555    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-tmp-dir\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:40:22.417679 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.417578    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-model-cache\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:40:22.419321 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.419300    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73b4621-5eb3-4893-afe3-673cc1cbfd53-kube-api-access-52rj9" (OuterVolumeSpecName: "kube-api-access-52rj9") pod "c73b4621-5eb3-4893-afe3-673cc1cbfd53" (UID: "c73b4621-5eb3-4893-afe3-673cc1cbfd53"). InnerVolumeSpecName "kube-api-access-52rj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:40:22.419508 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.419417    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c73b4621-5eb3-4893-afe3-673cc1cbfd53-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c73b4621-5eb3-4893-afe3-673cc1cbfd53" (UID: "c73b4621-5eb3-4893-afe3-673cc1cbfd53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:40:22.419508 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.419433    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-dshm" (OuterVolumeSpecName: "dshm") pod "c73b4621-5eb3-4893-afe3-673cc1cbfd53" (UID: "c73b4621-5eb3-4893-afe3-673cc1cbfd53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:40:22.519128 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.519026    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-home\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:40:22.519128 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.519060    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c73b4621-5eb3-4893-afe3-673cc1cbfd53-tls-certs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:40:22.519128 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.519074    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c73b4621-5eb3-4893-afe3-673cc1cbfd53-dshm\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:40:22.519128 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.519085    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52rj9\" (UniqueName: \"kubernetes.io/projected/c73b4621-5eb3-4893-afe3-673cc1cbfd53-kube-api-access-52rj9\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:40:22.578232 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.578204    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"]
Sep 06 19:40:22.580362 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.580342    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5ef25/rate-test-model-e5ef25-kserve-74fd7bb567-xbmxs"]
Sep 06 19:40:22.971651 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:22.971586    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp" podUID="7530715d-3155-4ab0-8e0c-4c9785ee470c" containerName="main" containerID="cri-o://00274852096eeeb5dd10cf358b71b23682760ef74015859697240378ee239334" gracePeriod=45
Sep 06 19:40:23.219125 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.219098    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:40:23.224587 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.224532    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-dshm\") pod \"7530715d-3155-4ab0-8e0c-4c9785ee470c\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") "
Sep 06 19:40:23.224587 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.224568    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-home\") pod \"7530715d-3155-4ab0-8e0c-4c9785ee470c\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") "
Sep 06 19:40:23.224780 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.224702    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-tmp-dir\") pod \"7530715d-3155-4ab0-8e0c-4c9785ee470c\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") "
Sep 06 19:40:23.224780 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.224737    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-model-cache\") pod \"7530715d-3155-4ab0-8e0c-4c9785ee470c\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") "
Sep 06 19:40:23.224896 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.224778    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgmjl\" (UniqueName: \"kubernetes.io/projected/7530715d-3155-4ab0-8e0c-4c9785ee470c-kube-api-access-bgmjl\") pod \"7530715d-3155-4ab0-8e0c-4c9785ee470c\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") "
Sep 06 19:40:23.224896 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.224816    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7530715d-3155-4ab0-8e0c-4c9785ee470c-tls-certs\") pod \"7530715d-3155-4ab0-8e0c-4c9785ee470c\" (UID: \"7530715d-3155-4ab0-8e0c-4c9785ee470c\") "
Sep 06 19:40:23.224896 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.224827    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-home" (OuterVolumeSpecName: "home") pod "7530715d-3155-4ab0-8e0c-4c9785ee470c" (UID: "7530715d-3155-4ab0-8e0c-4c9785ee470c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:40:23.225054 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.224962    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-model-cache" (OuterVolumeSpecName: "model-cache") pod "7530715d-3155-4ab0-8e0c-4c9785ee470c" (UID: "7530715d-3155-4ab0-8e0c-4c9785ee470c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:40:23.225054 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.224976    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7530715d-3155-4ab0-8e0c-4c9785ee470c" (UID: "7530715d-3155-4ab0-8e0c-4c9785ee470c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:40:23.225190 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.225165    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-tmp-dir\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:40:23.225265 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.225193    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-model-cache\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:40:23.225265 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.225206    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-home\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:40:23.226897 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.226871    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-dshm" (OuterVolumeSpecName: "dshm") pod "7530715d-3155-4ab0-8e0c-4c9785ee470c" (UID: "7530715d-3155-4ab0-8e0c-4c9785ee470c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:40:23.227013 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.226970    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7530715d-3155-4ab0-8e0c-4c9785ee470c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7530715d-3155-4ab0-8e0c-4c9785ee470c" (UID: "7530715d-3155-4ab0-8e0c-4c9785ee470c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:40:23.227013 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.226978    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7530715d-3155-4ab0-8e0c-4c9785ee470c-kube-api-access-bgmjl" (OuterVolumeSpecName: "kube-api-access-bgmjl") pod "7530715d-3155-4ab0-8e0c-4c9785ee470c" (UID: "7530715d-3155-4ab0-8e0c-4c9785ee470c"). InnerVolumeSpecName "kube-api-access-bgmjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:40:23.264040 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.264010    2566 generic.go:356] "Generic (PLEG): container finished" podID="7530715d-3155-4ab0-8e0c-4c9785ee470c" containerID="00274852096eeeb5dd10cf358b71b23682760ef74015859697240378ee239334" exitCode=0
Sep 06 19:40:23.264161 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.264054    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"
Sep 06 19:40:23.264161 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.264094    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp" event={"ID":"7530715d-3155-4ab0-8e0c-4c9785ee470c","Type":"ContainerDied","Data":"00274852096eeeb5dd10cf358b71b23682760ef74015859697240378ee239334"}
Sep 06 19:40:23.264161 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.264141    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp" event={"ID":"7530715d-3155-4ab0-8e0c-4c9785ee470c","Type":"ContainerDied","Data":"3f6f4f50bead0afa42e5f9cdd71b167e67efe7675f274655e380472b487d8955"}
Sep 06 19:40:23.264161 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.264158    2566 scope.go:117] "RemoveContainer" containerID="00274852096eeeb5dd10cf358b71b23682760ef74015859697240378ee239334"
Sep 06 19:40:23.273581 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.273566    2566 scope.go:117] "RemoveContainer" containerID="00274852096eeeb5dd10cf358b71b23682760ef74015859697240378ee239334"
Sep 06 19:40:23.273888 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:40:23.273871    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00274852096eeeb5dd10cf358b71b23682760ef74015859697240378ee239334\": container with ID starting with 00274852096eeeb5dd10cf358b71b23682760ef74015859697240378ee239334 not found: ID does not exist" containerID="00274852096eeeb5dd10cf358b71b23682760ef74015859697240378ee239334"
Sep 06 19:40:23.273960 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.273897    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00274852096eeeb5dd10cf358b71b23682760ef74015859697240378ee239334"} err="failed to get container status \"00274852096eeeb5dd10cf358b71b23682760ef74015859697240378ee239334\": rpc error: code = NotFound desc = could not find container \"00274852096eeeb5dd10cf358b71b23682760ef74015859697240378ee239334\": container with ID starting with 00274852096eeeb5dd10cf358b71b23682760ef74015859697240378ee239334 not found: ID does not exist"
Sep 06 19:40:23.283949 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.283928    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"]
Sep 06 19:40:23.289422 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.289402    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-476f98/rate-test-model-476f98-kserve-bd658d75-t2vsp"]
Sep 06 19:40:23.326089 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.326062    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgmjl\" (UniqueName: \"kubernetes.io/projected/7530715d-3155-4ab0-8e0c-4c9785ee470c-kube-api-access-bgmjl\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:40:23.326089 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.326084    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7530715d-3155-4ab0-8e0c-4c9785ee470c-tls-certs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:40:23.326268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:23.326097    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7530715d-3155-4ab0-8e0c-4c9785ee470c-dshm\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:40:24.309681 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:24.309647    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7530715d-3155-4ab0-8e0c-4c9785ee470c" path="/var/lib/kubelet/pods/7530715d-3155-4ab0-8e0c-4c9785ee470c/volumes"
Sep 06 19:40:24.310032 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:24.310018    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c73b4621-5eb3-4893-afe3-673cc1cbfd53" path="/var/lib/kubelet/pods/c73b4621-5eb3-4893-afe3-673cc1cbfd53/volumes"
Sep 06 19:40:42.153637 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.153585    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"]
Sep 06 19:40:42.154205 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.154189    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c73b4621-5eb3-4893-afe3-673cc1cbfd53" containerName="main"
Sep 06 19:40:42.154253 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.154209    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c73b4621-5eb3-4893-afe3-673cc1cbfd53" containerName="main"
Sep 06 19:40:42.154253 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.154237    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7530715d-3155-4ab0-8e0c-4c9785ee470c" containerName="main"
Sep 06 19:40:42.154253 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.154245    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7530715d-3155-4ab0-8e0c-4c9785ee470c" containerName="main"
Sep 06 19:40:42.154359 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.154328    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c73b4621-5eb3-4893-afe3-673cc1cbfd53" containerName="main"
Sep 06 19:40:42.154359 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.154337    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7530715d-3155-4ab0-8e0c-4c9785ee470c" containerName="main"
Sep 06 19:40:42.157886 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.157858    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"
Sep 06 19:40:42.160257 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.160233    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-88ac22-serving-cert\""
Sep 06 19:40:42.164671 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.164646    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"]
Sep 06 19:40:42.288256 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.288223    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd1327d2-92f9-4270-8979-6a1eab5c0586-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m\" (UID: \"cd1327d2-92f9-4270-8979-6a1eab5c0586\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"
Sep 06 19:40:42.288424 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.288276    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd1327d2-92f9-4270-8979-6a1eab5c0586-maas-api-tls\") pod \"maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m\" (UID: \"cd1327d2-92f9-4270-8979-6a1eab5c0586\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"
Sep 06 19:40:42.288424 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.288360    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fz26\" (UniqueName: \"kubernetes.io/projected/cd1327d2-92f9-4270-8979-6a1eab5c0586-kube-api-access-2fz26\") pod \"maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m\" (UID: \"cd1327d2-92f9-4270-8979-6a1eab5c0586\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"
Sep 06 19:40:42.288424 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.288419    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd1327d2-92f9-4270-8979-6a1eab5c0586-metrics-certs\") pod \"maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m\" (UID: \"cd1327d2-92f9-4270-8979-6a1eab5c0586\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"
Sep 06 19:40:42.389638 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.389591    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fz26\" (UniqueName: \"kubernetes.io/projected/cd1327d2-92f9-4270-8979-6a1eab5c0586-kube-api-access-2fz26\") pod \"maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m\" (UID: \"cd1327d2-92f9-4270-8979-6a1eab5c0586\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"
Sep 06 19:40:42.389829 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.389651    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd1327d2-92f9-4270-8979-6a1eab5c0586-metrics-certs\") pod \"maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m\" (UID: \"cd1327d2-92f9-4270-8979-6a1eab5c0586\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"
Sep 06 19:40:42.389901 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.389872    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd1327d2-92f9-4270-8979-6a1eab5c0586-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m\" (UID: \"cd1327d2-92f9-4270-8979-6a1eab5c0586\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"
Sep 06 19:40:42.389987 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.389938    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd1327d2-92f9-4270-8979-6a1eab5c0586-maas-api-tls\") pod \"maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m\" (UID: \"cd1327d2-92f9-4270-8979-6a1eab5c0586\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"
Sep 06 19:40:42.390084 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:40:42.390067    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-serving-cert: secret "maas-api-e2e-ipp-b-88ac22-serving-cert" not found
Sep 06 19:40:42.390171 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:40:42.390147    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd1327d2-92f9-4270-8979-6a1eab5c0586-maas-api-tls podName:cd1327d2-92f9-4270-8979-6a1eab5c0586 nodeName:}" failed. No retries permitted until 2026-09-06 19:40:42.890124277 +0000 UTC m=+1777.071306061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cd1327d2-92f9-4270-8979-6a1eab5c0586-maas-api-tls") pod "maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m" (UID: "cd1327d2-92f9-4270-8979-6a1eab5c0586") : secret "maas-api-e2e-ipp-b-88ac22-serving-cert" not found
Sep 06 19:40:42.390522 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.390504    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd1327d2-92f9-4270-8979-6a1eab5c0586-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m\" (UID: \"cd1327d2-92f9-4270-8979-6a1eab5c0586\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"
Sep 06 19:40:42.392329 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.392313    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd1327d2-92f9-4270-8979-6a1eab5c0586-metrics-certs\") pod \"maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m\" (UID: \"cd1327d2-92f9-4270-8979-6a1eab5c0586\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"
Sep 06 19:40:42.397959 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.397934    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fz26\" (UniqueName: \"kubernetes.io/projected/cd1327d2-92f9-4270-8979-6a1eab5c0586-kube-api-access-2fz26\") pod \"maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m\" (UID: \"cd1327d2-92f9-4270-8979-6a1eab5c0586\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"
Sep 06 19:40:42.892705 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.892665    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2"]
Sep 06 19:40:42.894164 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.894130    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd1327d2-92f9-4270-8979-6a1eab5c0586-maas-api-tls\") pod \"maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m\" (UID: \"cd1327d2-92f9-4270-8979-6a1eab5c0586\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"
Sep 06 19:40:42.896687 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.896656    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2"
Sep 06 19:40:42.896959 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.896926    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd1327d2-92f9-4270-8979-6a1eab5c0586-maas-api-tls\") pod \"maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m\" (UID: \"cd1327d2-92f9-4270-8979-6a1eab5c0586\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"
Sep 06 19:40:42.899126 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.899107    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-88ac22-dockercfg-wcvbh\""
Sep 06 19:40:42.899233 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.899132    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-88ac22\""
Sep 06 19:40:42.906723 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.906703    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2"]
Sep 06 19:40:42.994857 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.994832    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bef79f-1e1b-4549-9f68-f984b295e10b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2\" (UID: \"76bef79f-1e1b-4549-9f68-f984b295e10b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2"
Sep 06 19:40:42.994997 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:42.994945    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjg7l\" (UniqueName: \"kubernetes.io/projected/76bef79f-1e1b-4549-9f68-f984b295e10b-kube-api-access-qjg7l\") pod \"payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2\" (UID: \"76bef79f-1e1b-4549-9f68-f984b295e10b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2"
Sep 06 19:40:43.071946 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:43.071916    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"
Sep 06 19:40:43.096144 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:43.096116    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bef79f-1e1b-4549-9f68-f984b295e10b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2\" (UID: \"76bef79f-1e1b-4549-9f68-f984b295e10b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2"
Sep 06 19:40:43.096259 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:43.096220    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjg7l\" (UniqueName: \"kubernetes.io/projected/76bef79f-1e1b-4549-9f68-f984b295e10b-kube-api-access-qjg7l\") pod \"payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2\" (UID: \"76bef79f-1e1b-4549-9f68-f984b295e10b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2"
Sep 06 19:40:43.097159 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:43.097133    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bef79f-1e1b-4549-9f68-f984b295e10b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2\" (UID: \"76bef79f-1e1b-4549-9f68-f984b295e10b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2"
Sep 06 19:40:43.104703 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:43.104677    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjg7l\" (UniqueName: \"kubernetes.io/projected/76bef79f-1e1b-4549-9f68-f984b295e10b-kube-api-access-qjg7l\") pod \"payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2\" (UID: \"76bef79f-1e1b-4549-9f68-f984b295e10b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2"
Sep 06 19:40:43.215365 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:43.215336    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2"
Sep 06 19:40:43.342656 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:43.342636    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"]
Sep 06 19:40:43.343269 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:40:43.343246    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1327d2_92f9_4270_8979_6a1eab5c0586.slice/crio-0d5d3166a4b52868ed1411d3821324ad303b3934e18fc481b80d8f806f7ed69c WatchSource:0}: Error finding container 0d5d3166a4b52868ed1411d3821324ad303b3934e18fc481b80d8f806f7ed69c: Status 404 returned error can't find the container with id 0d5d3166a4b52868ed1411d3821324ad303b3934e18fc481b80d8f806f7ed69c
Sep 06 19:40:43.376552 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:43.376525    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m" event={"ID":"cd1327d2-92f9-4270-8979-6a1eab5c0586","Type":"ContainerStarted","Data":"0d5d3166a4b52868ed1411d3821324ad303b3934e18fc481b80d8f806f7ed69c"}
Sep 06 19:40:43.487358 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:43.487322    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2"]
Sep 06 19:40:43.488727 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:40:43.488702    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bef79f_1e1b_4549_9f68_f984b295e10b.slice/crio-7ebafd02555b6bd590c05c65d0911df1a57d68822fb560b26414123503119d42 WatchSource:0}: Error finding container 7ebafd02555b6bd590c05c65d0911df1a57d68822fb560b26414123503119d42: Status 404 returned error can't find the container with id 7ebafd02555b6bd590c05c65d0911df1a57d68822fb560b26414123503119d42
Sep 06 19:40:44.383736 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:44.383701    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2" event={"ID":"76bef79f-1e1b-4549-9f68-f984b295e10b","Type":"ContainerStarted","Data":"418d042d7bbe7bcbf9f2e47d1587b469e132177b4a491362ed20f9fbe81911b5"}
Sep 06 19:40:44.384208 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:44.383759    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2" event={"ID":"76bef79f-1e1b-4549-9f68-f984b295e10b","Type":"ContainerStarted","Data":"7ebafd02555b6bd590c05c65d0911df1a57d68822fb560b26414123503119d42"}
Sep 06 19:40:44.384208 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:44.383903    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2"
Sep 06 19:40:44.385459 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:44.385435    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m" event={"ID":"cd1327d2-92f9-4270-8979-6a1eab5c0586","Type":"ContainerStarted","Data":"1c730303cdda6dd63da44fbb91438c3a9e0dc46aba1b7b4bbd304f4fb3db4f63"}
Sep 06 19:40:44.385590 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:44.385492    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"
Sep 06 19:40:44.400414 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:44.400366    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2" podStartSLOduration=2.400353772 podStartE2EDuration="2.400353772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:40:44.398580975 +0000 UTC m=+1778.579762783" watchObservedRunningTime="2026-09-06 19:40:44.400353772 +0000 UTC m=+1778.581535578"
Sep 06 19:40:44.415083 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:44.415043    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m" podStartSLOduration=2.02189754 podStartE2EDuration="2.415031158s" totalImagesPullingTime="393.133618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:40:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:40:44.413464813 +0000 UTC m=+1778.594646617" watchObservedRunningTime="2026-09-06 19:40:44.415031158 +0000 UTC m=+1778.596212966"
Sep 06 19:40:47.940771 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:47.940733    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:40:50.396540 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:50.396509    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"
Sep 06 19:40:55.393783 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:40:55.393742    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2"
Sep 06 19:41:06.468458 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:06.468433    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnssq_9254e6f1-b73d-49cc-8bc1-513925ccde4f/ovn-acl-logging/0.log"
Sep 06 19:41:06.473117 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:06.473099    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnssq_9254e6f1-b73d-49cc-8bc1-513925ccde4f/ovn-acl-logging/0.log"
Sep 06 19:41:10.985225 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:10.985196    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2_76bef79f-1e1b-4549-9f68-f984b295e10b/payload-processing/0.log"
Sep 06 19:41:22.246343 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.246310    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"]
Sep 06 19:41:22.249855 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.249837    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:22.253145 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.253127    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-410f4d\"/\"openshift-service-ca.crt\""
Sep 06 19:41:22.253268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.253203    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-410f4d\"/\"kube-root-ca.crt\""
Sep 06 19:41:22.253268 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.253208    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-410f4d\"/\"ipp-route-410f4d-kserve-self-signed-certs\""
Sep 06 19:41:22.253370 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.253207    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-410f4d\"/\"default-dockercfg-l78cr\""
Sep 06 19:41:22.260142 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.260115    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"]
Sep 06 19:41:22.350953 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.350927    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-tmp-dir\") pod \"ipp-route-410f4d-kserve-76d89fb5f-nwc8g\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") " pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:22.351098 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.350960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-home\") pod \"ipp-route-410f4d-kserve-76d89fb5f-nwc8g\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") " pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:22.351098 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.351042    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-model-cache\") pod \"ipp-route-410f4d-kserve-76d89fb5f-nwc8g\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") " pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:22.351098 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.351073    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-tls-certs\") pod \"ipp-route-410f4d-kserve-76d89fb5f-nwc8g\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") " pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:22.351213 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.351099    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-dshm\") pod \"ipp-route-410f4d-kserve-76d89fb5f-nwc8g\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") " pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:22.351213 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.351131    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbdfd\" (UniqueName: \"kubernetes.io/projected/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-kube-api-access-cbdfd\") pod \"ipp-route-410f4d-kserve-76d89fb5f-nwc8g\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") " pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:22.451709 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.451683    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-home\") pod \"ipp-route-410f4d-kserve-76d89fb5f-nwc8g\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") " pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:22.451863 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.451736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-model-cache\") pod \"ipp-route-410f4d-kserve-76d89fb5f-nwc8g\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") " pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:22.451863 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.451764    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-tls-certs\") pod \"ipp-route-410f4d-kserve-76d89fb5f-nwc8g\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") " pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:22.451863 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.451783    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-dshm\") pod \"ipp-route-410f4d-kserve-76d89fb5f-nwc8g\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") " pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:22.451863 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.451803    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbdfd\" (UniqueName: \"kubernetes.io/projected/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-kube-api-access-cbdfd\") pod \"ipp-route-410f4d-kserve-76d89fb5f-nwc8g\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") " pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:22.452039 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.452016    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-tmp-dir\") pod \"ipp-route-410f4d-kserve-76d89fb5f-nwc8g\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") " pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:22.452161 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.452135    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-home\") pod \"ipp-route-410f4d-kserve-76d89fb5f-nwc8g\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") " pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:22.452161 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.452149    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-model-cache\") pod \"ipp-route-410f4d-kserve-76d89fb5f-nwc8g\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") " pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:22.452346 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.452327    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-tmp-dir\") pod \"ipp-route-410f4d-kserve-76d89fb5f-nwc8g\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") " pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:22.454144 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.454123    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-dshm\") pod \"ipp-route-410f4d-kserve-76d89fb5f-nwc8g\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") " pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:22.454377 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.454361    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-tls-certs\") pod \"ipp-route-410f4d-kserve-76d89fb5f-nwc8g\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") " pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:22.459228 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.459202    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbdfd\" (UniqueName: \"kubernetes.io/projected/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-kube-api-access-cbdfd\") pod \"ipp-route-410f4d-kserve-76d89fb5f-nwc8g\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") " pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:22.560820 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.560736    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:22.837071 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:22.837043    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"]
Sep 06 19:41:22.838655 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:41:22.838629    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c9aeeaf_9e6a_4a8f_8605_131329bda74d.slice/crio-d3751943fd89ef335f1d63fc30c1231d2214be6289f253578d6f2ec42f39a71c WatchSource:0}: Error finding container d3751943fd89ef335f1d63fc30c1231d2214be6289f253578d6f2ec42f39a71c: Status 404 returned error can't find the container with id d3751943fd89ef335f1d63fc30c1231d2214be6289f253578d6f2ec42f39a71c
Sep 06 19:41:23.603838 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:23.603795    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g" event={"ID":"8c9aeeaf-9e6a-4a8f-8605-131329bda74d","Type":"ContainerStarted","Data":"30e3a832800f0cc42682ee444928ec83ce9220bc0721b25fcc3e1f07a7d2a4fe"}
Sep 06 19:41:23.604233 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:23.603858    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g" event={"ID":"8c9aeeaf-9e6a-4a8f-8605-131329bda74d","Type":"ContainerStarted","Data":"d3751943fd89ef335f1d63fc30c1231d2214be6289f253578d6f2ec42f39a71c"}
Sep 06 19:41:23.619700 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:23.619650    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g" podStartSLOduration=1.619635879 podStartE2EDuration="1.619635879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:41:23.618311402 +0000 UTC m=+1817.799493209" watchObservedRunningTime="2026-09-06 19:41:23.619635879 +0000 UTC m=+1817.800817680"
Sep 06 19:41:26.952563 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:26.952519    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:41:32.561023 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:32.560989    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:32.561023 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:32.561033    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:32.573422 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:32.573397    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:32.667366 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:32.667331    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:41:42.350353 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:42.350318    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:41:48.816836 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:48.816806    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2_76bef79f-1e1b-4549-9f68-f984b295e10b/payload-processing/0.log"
Sep 06 19:41:54.243020 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:54.242985    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld"]
Sep 06 19:41:54.246713 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:54.246695    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld"
Sep 06 19:41:54.249040 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:54.249018    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-413b7f-dockercfg-nvwh5\""
Sep 06 19:41:54.249386 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:54.249364    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-413b7f\""
Sep 06 19:41:54.256561 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:54.256537    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld"]
Sep 06 19:41:54.441768 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:54.441736    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06e05ac3-0923-4d7a-a2e3-7c426db583f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld\" (UID: \"06e05ac3-0923-4d7a-a2e3-7c426db583f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld"
Sep 06 19:41:54.441930 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:54.441808    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnxgz\" (UniqueName: \"kubernetes.io/projected/06e05ac3-0923-4d7a-a2e3-7c426db583f2-kube-api-access-lnxgz\") pod \"payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld\" (UID: \"06e05ac3-0923-4d7a-a2e3-7c426db583f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld"
Sep 06 19:41:54.542815 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:54.542730    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnxgz\" (UniqueName: \"kubernetes.io/projected/06e05ac3-0923-4d7a-a2e3-7c426db583f2-kube-api-access-lnxgz\") pod \"payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld\" (UID: \"06e05ac3-0923-4d7a-a2e3-7c426db583f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld"
Sep 06 19:41:54.542815 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:54.542796    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06e05ac3-0923-4d7a-a2e3-7c426db583f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld\" (UID: \"06e05ac3-0923-4d7a-a2e3-7c426db583f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld"
Sep 06 19:41:54.543424 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:54.543403    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06e05ac3-0923-4d7a-a2e3-7c426db583f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld\" (UID: \"06e05ac3-0923-4d7a-a2e3-7c426db583f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld"
Sep 06 19:41:54.551288 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:54.551200    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnxgz\" (UniqueName: \"kubernetes.io/projected/06e05ac3-0923-4d7a-a2e3-7c426db583f2-kube-api-access-lnxgz\") pod \"payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld\" (UID: \"06e05ac3-0923-4d7a-a2e3-7c426db583f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld"
Sep 06 19:41:54.558375 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:54.558354    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld"
Sep 06 19:41:54.830030 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:54.830001    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld"]
Sep 06 19:41:54.831150 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:41:54.831128    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e05ac3_0923_4d7a_a2e3_7c426db583f2.slice/crio-50d6178caa846a51807ee82bc966dda624107b6051a57b50e40d028b5c898002 WatchSource:0}: Error finding container 50d6178caa846a51807ee82bc966dda624107b6051a57b50e40d028b5c898002: Status 404 returned error can't find the container with id 50d6178caa846a51807ee82bc966dda624107b6051a57b50e40d028b5c898002
Sep 06 19:41:55.785267 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:55.785227    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld" event={"ID":"06e05ac3-0923-4d7a-a2e3-7c426db583f2","Type":"ContainerStarted","Data":"d7d48ede337045a932e639c46cc48b3f1591145ca5acb4209dffb7765625661e"}
Sep 06 19:41:55.785681 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:55.785295    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld"
Sep 06 19:41:55.785681 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:55.785317    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld" event={"ID":"06e05ac3-0923-4d7a-a2e3-7c426db583f2","Type":"ContainerStarted","Data":"50d6178caa846a51807ee82bc966dda624107b6051a57b50e40d028b5c898002"}
Sep 06 19:41:55.800653 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:41:55.800579    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld" podStartSLOduration=1.8005642480000001 podStartE2EDuration="1.800564248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:41:55.79936653 +0000 UTC m=+1849.980548335" watchObservedRunningTime="2026-09-06 19:41:55.800564248 +0000 UTC m=+1849.981746101"
Sep 06 19:42:01.652544 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:01.652510    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:42:06.793172 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:06.793143    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld"
Sep 06 19:42:24.091486 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:24.091452    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld"]
Sep 06 19:42:24.092041 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:24.091714    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld" podUID="06e05ac3-0923-4d7a-a2e3-7c426db583f2" containerName="payload-pre-processing" containerID="cri-o://d7d48ede337045a932e639c46cc48b3f1591145ca5acb4209dffb7765625661e" gracePeriod=30
Sep 06 19:42:24.343114 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:24.343062    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld"
Sep 06 19:42:24.498144 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:24.498115    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06e05ac3-0923-4d7a-a2e3-7c426db583f2-plugins-config-volume\") pod \"06e05ac3-0923-4d7a-a2e3-7c426db583f2\" (UID: \"06e05ac3-0923-4d7a-a2e3-7c426db583f2\") "
Sep 06 19:42:24.498323 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:24.498245    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnxgz\" (UniqueName: \"kubernetes.io/projected/06e05ac3-0923-4d7a-a2e3-7c426db583f2-kube-api-access-lnxgz\") pod \"06e05ac3-0923-4d7a-a2e3-7c426db583f2\" (UID: \"06e05ac3-0923-4d7a-a2e3-7c426db583f2\") "
Sep 06 19:42:24.498557 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:24.498534    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e05ac3-0923-4d7a-a2e3-7c426db583f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "06e05ac3-0923-4d7a-a2e3-7c426db583f2" (UID: "06e05ac3-0923-4d7a-a2e3-7c426db583f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:42:24.500880 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:24.500857    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e05ac3-0923-4d7a-a2e3-7c426db583f2-kube-api-access-lnxgz" (OuterVolumeSpecName: "kube-api-access-lnxgz") pod "06e05ac3-0923-4d7a-a2e3-7c426db583f2" (UID: "06e05ac3-0923-4d7a-a2e3-7c426db583f2"). InnerVolumeSpecName "kube-api-access-lnxgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:42:24.599933 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:24.599873    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnxgz\" (UniqueName: \"kubernetes.io/projected/06e05ac3-0923-4d7a-a2e3-7c426db583f2-kube-api-access-lnxgz\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:42:24.599933 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:24.599897    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06e05ac3-0923-4d7a-a2e3-7c426db583f2-plugins-config-volume\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:42:24.948035 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:24.947957    2566 generic.go:356] "Generic (PLEG): container finished" podID="06e05ac3-0923-4d7a-a2e3-7c426db583f2" containerID="d7d48ede337045a932e639c46cc48b3f1591145ca5acb4209dffb7765625661e" exitCode=0
Sep 06 19:42:24.948035 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:24.948006    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld"
Sep 06 19:42:24.948205 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:24.948038    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld" event={"ID":"06e05ac3-0923-4d7a-a2e3-7c426db583f2","Type":"ContainerDied","Data":"d7d48ede337045a932e639c46cc48b3f1591145ca5acb4209dffb7765625661e"}
Sep 06 19:42:24.948205 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:24.948090    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld" event={"ID":"06e05ac3-0923-4d7a-a2e3-7c426db583f2","Type":"ContainerDied","Data":"50d6178caa846a51807ee82bc966dda624107b6051a57b50e40d028b5c898002"}
Sep 06 19:42:24.948205 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:24.948111    2566 scope.go:117] "RemoveContainer" containerID="d7d48ede337045a932e639c46cc48b3f1591145ca5acb4209dffb7765625661e"
Sep 06 19:42:24.957823 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:24.957806    2566 scope.go:117] "RemoveContainer" containerID="d7d48ede337045a932e639c46cc48b3f1591145ca5acb4209dffb7765625661e"
Sep 06 19:42:24.958127 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:42:24.958105    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7d48ede337045a932e639c46cc48b3f1591145ca5acb4209dffb7765625661e\": container with ID starting with d7d48ede337045a932e639c46cc48b3f1591145ca5acb4209dffb7765625661e not found: ID does not exist" containerID="d7d48ede337045a932e639c46cc48b3f1591145ca5acb4209dffb7765625661e"
Sep 06 19:42:24.958216 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:24.958134    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d48ede337045a932e639c46cc48b3f1591145ca5acb4209dffb7765625661e"} err="failed to get container status \"d7d48ede337045a932e639c46cc48b3f1591145ca5acb4209dffb7765625661e\": rpc error: code = NotFound desc = could not find container \"d7d48ede337045a932e639c46cc48b3f1591145ca5acb4209dffb7765625661e\": container with ID starting with d7d48ede337045a932e639c46cc48b3f1591145ca5acb4209dffb7765625661e not found: ID does not exist"
Sep 06 19:42:24.968745 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:24.968720    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld"]
Sep 06 19:42:24.972437 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:24.972418    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-413b7f-686f7bc894-gd4ld"]
Sep 06 19:42:26.310769 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:26.310734    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06e05ac3-0923-4d7a-a2e3-7c426db583f2" path="/var/lib/kubelet/pods/06e05ac3-0923-4d7a-a2e3-7c426db583f2/volumes"
Sep 06 19:42:28.844282 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:28.844204    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:42:50.155047 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:50.155012    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:42:55.089289 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:55.089254    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"]
Sep 06 19:42:59.844122 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:42:59.844086    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:43:10.140754 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:10.140690    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g" podUID="8c9aeeaf-9e6a-4a8f-8605-131329bda74d" containerName="main" containerID="cri-o://30e3a832800f0cc42682ee444928ec83ce9220bc0721b25fcc3e1f07a7d2a4fe" gracePeriod=45
Sep 06 19:43:10.394138 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:10.394078    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:43:10.488524 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:10.488495    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbdfd\" (UniqueName: \"kubernetes.io/projected/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-kube-api-access-cbdfd\") pod \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") "
Sep 06 19:43:10.488724 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:10.488559    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-dshm\") pod \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") "
Sep 06 19:43:10.488724 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:10.488598    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-home\") pod \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") "
Sep 06 19:43:10.488724 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:10.488664    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-tmp-dir\") pod \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") "
Sep 06 19:43:10.488724 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:10.488690    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-model-cache\") pod \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") "
Sep 06 19:43:10.488950 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:10.488740    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-tls-certs\") pod \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\" (UID: \"8c9aeeaf-9e6a-4a8f-8605-131329bda74d\") "
Sep 06 19:43:10.488950 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:10.488889    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-home" (OuterVolumeSpecName: "home") pod "8c9aeeaf-9e6a-4a8f-8605-131329bda74d" (UID: "8c9aeeaf-9e6a-4a8f-8605-131329bda74d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:10.489043 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:10.488979    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8c9aeeaf-9e6a-4a8f-8605-131329bda74d" (UID: "8c9aeeaf-9e6a-4a8f-8605-131329bda74d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:10.489043 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:10.488990    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c9aeeaf-9e6a-4a8f-8605-131329bda74d" (UID: "8c9aeeaf-9e6a-4a8f-8605-131329bda74d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:10.489139 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:10.489087    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-home\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:43:10.489139 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:10.489105    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-tmp-dir\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:43:10.489139 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:10.489117    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-model-cache\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:43:10.490897 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:10.490874    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-dshm" (OuterVolumeSpecName: "dshm") pod "8c9aeeaf-9e6a-4a8f-8605-131329bda74d" (UID: "8c9aeeaf-9e6a-4a8f-8605-131329bda74d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:10.490897 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:10.490890    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c9aeeaf-9e6a-4a8f-8605-131329bda74d" (UID: "8c9aeeaf-9e6a-4a8f-8605-131329bda74d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:43:10.491074 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:10.491002    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-kube-api-access-cbdfd" (OuterVolumeSpecName: "kube-api-access-cbdfd") pod "8c9aeeaf-9e6a-4a8f-8605-131329bda74d" (UID: "8c9aeeaf-9e6a-4a8f-8605-131329bda74d"). InnerVolumeSpecName "kube-api-access-cbdfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:43:10.589925 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:10.589898    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-tls-certs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:43:10.589925 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:10.589921    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbdfd\" (UniqueName: \"kubernetes.io/projected/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-kube-api-access-cbdfd\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:43:10.590072 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:10.589933    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c9aeeaf-9e6a-4a8f-8605-131329bda74d-dshm\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:43:11.216708 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:11.216669    2566 generic.go:356] "Generic (PLEG): container finished" podID="8c9aeeaf-9e6a-4a8f-8605-131329bda74d" containerID="30e3a832800f0cc42682ee444928ec83ce9220bc0721b25fcc3e1f07a7d2a4fe" exitCode=0
Sep 06 19:43:11.217079 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:11.216721    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"
Sep 06 19:43:11.217079 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:11.216760    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g" event={"ID":"8c9aeeaf-9e6a-4a8f-8605-131329bda74d","Type":"ContainerDied","Data":"30e3a832800f0cc42682ee444928ec83ce9220bc0721b25fcc3e1f07a7d2a4fe"}
Sep 06 19:43:11.217079 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:11.216805    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g" event={"ID":"8c9aeeaf-9e6a-4a8f-8605-131329bda74d","Type":"ContainerDied","Data":"d3751943fd89ef335f1d63fc30c1231d2214be6289f253578d6f2ec42f39a71c"}
Sep 06 19:43:11.217079 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:11.216825    2566 scope.go:117] "RemoveContainer" containerID="30e3a832800f0cc42682ee444928ec83ce9220bc0721b25fcc3e1f07a7d2a4fe"
Sep 06 19:43:11.234779 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:11.234762    2566 scope.go:117] "RemoveContainer" containerID="30e3a832800f0cc42682ee444928ec83ce9220bc0721b25fcc3e1f07a7d2a4fe"
Sep 06 19:43:11.235031 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:43:11.235012    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e3a832800f0cc42682ee444928ec83ce9220bc0721b25fcc3e1f07a7d2a4fe\": container with ID starting with 30e3a832800f0cc42682ee444928ec83ce9220bc0721b25fcc3e1f07a7d2a4fe not found: ID does not exist" containerID="30e3a832800f0cc42682ee444928ec83ce9220bc0721b25fcc3e1f07a7d2a4fe"
Sep 06 19:43:11.235089 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:11.235039    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e3a832800f0cc42682ee444928ec83ce9220bc0721b25fcc3e1f07a7d2a4fe"} err="failed to get container status \"30e3a832800f0cc42682ee444928ec83ce9220bc0721b25fcc3e1f07a7d2a4fe\": rpc error: code = NotFound desc = could not find container \"30e3a832800f0cc42682ee444928ec83ce9220bc0721b25fcc3e1f07a7d2a4fe\": container with ID starting with 30e3a832800f0cc42682ee444928ec83ce9220bc0721b25fcc3e1f07a7d2a4fe not found: ID does not exist"
Sep 06 19:43:11.242426 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:11.242405    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"]
Sep 06 19:43:11.245841 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:11.245820    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-410f4d/ipp-route-410f4d-kserve-76d89fb5f-nwc8g"]
Sep 06 19:43:12.311192 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:12.311164    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c9aeeaf-9e6a-4a8f-8605-131329bda74d" path="/var/lib/kubelet/pods/8c9aeeaf-9e6a-4a8f-8605-131329bda74d/volumes"
Sep 06 19:43:48.431284 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.431254    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"]
Sep 06 19:43:48.431770 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.431452    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m" podUID="cd1327d2-92f9-4270-8979-6a1eab5c0586" containerName="maas-api" containerID="cri-o://1c730303cdda6dd63da44fbb91438c3a9e0dc46aba1b7b4bbd304f4fb3db4f63" gracePeriod=30
Sep 06 19:43:48.486737 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.486705    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2"]
Sep 06 19:43:48.487515 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.487457    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2" podUID="76bef79f-1e1b-4549-9f68-f984b295e10b" containerName="payload-processing" containerID="cri-o://418d042d7bbe7bcbf9f2e47d1587b469e132177b4a491362ed20f9fbe81911b5" gracePeriod=30
Sep 06 19:43:48.733618 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.733574    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"
Sep 06 19:43:48.762849 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.762827    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2"
Sep 06 19:43:48.816788 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.816764    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd1327d2-92f9-4270-8979-6a1eab5c0586-metrics-certs\") pod \"cd1327d2-92f9-4270-8979-6a1eab5c0586\" (UID: \"cd1327d2-92f9-4270-8979-6a1eab5c0586\") "
Sep 06 19:43:48.816931 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.816804    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd1327d2-92f9-4270-8979-6a1eab5c0586-maas-api-tls\") pod \"cd1327d2-92f9-4270-8979-6a1eab5c0586\" (UID: \"cd1327d2-92f9-4270-8979-6a1eab5c0586\") "
Sep 06 19:43:48.816931 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.816844    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bef79f-1e1b-4549-9f68-f984b295e10b-plugins-config-volume\") pod \"76bef79f-1e1b-4549-9f68-f984b295e10b\" (UID: \"76bef79f-1e1b-4549-9f68-f984b295e10b\") "
Sep 06 19:43:48.816931 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.816895    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd1327d2-92f9-4270-8979-6a1eab5c0586-openshift-service-ca\") pod \"cd1327d2-92f9-4270-8979-6a1eab5c0586\" (UID: \"cd1327d2-92f9-4270-8979-6a1eab5c0586\") "
Sep 06 19:43:48.816931 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.816926    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjg7l\" (UniqueName: \"kubernetes.io/projected/76bef79f-1e1b-4549-9f68-f984b295e10b-kube-api-access-qjg7l\") pod \"76bef79f-1e1b-4549-9f68-f984b295e10b\" (UID: \"76bef79f-1e1b-4549-9f68-f984b295e10b\") "
Sep 06 19:43:48.817151 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.816969    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fz26\" (UniqueName: \"kubernetes.io/projected/cd1327d2-92f9-4270-8979-6a1eab5c0586-kube-api-access-2fz26\") pod \"cd1327d2-92f9-4270-8979-6a1eab5c0586\" (UID: \"cd1327d2-92f9-4270-8979-6a1eab5c0586\") "
Sep 06 19:43:48.817281 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.817241    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd1327d2-92f9-4270-8979-6a1eab5c0586-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cd1327d2-92f9-4270-8979-6a1eab5c0586" (UID: "cd1327d2-92f9-4270-8979-6a1eab5c0586"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:43:48.817341 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.817268    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76bef79f-1e1b-4549-9f68-f984b295e10b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "76bef79f-1e1b-4549-9f68-f984b295e10b" (UID: "76bef79f-1e1b-4549-9f68-f984b295e10b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:43:48.819105 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.819078    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd1327d2-92f9-4270-8979-6a1eab5c0586-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cd1327d2-92f9-4270-8979-6a1eab5c0586" (UID: "cd1327d2-92f9-4270-8979-6a1eab5c0586"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:43:48.819217 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.819144    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bef79f-1e1b-4549-9f68-f984b295e10b-kube-api-access-qjg7l" (OuterVolumeSpecName: "kube-api-access-qjg7l") pod "76bef79f-1e1b-4549-9f68-f984b295e10b" (UID: "76bef79f-1e1b-4549-9f68-f984b295e10b"). InnerVolumeSpecName "kube-api-access-qjg7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:43:48.819217 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.819161    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1327d2-92f9-4270-8979-6a1eab5c0586-kube-api-access-2fz26" (OuterVolumeSpecName: "kube-api-access-2fz26") pod "cd1327d2-92f9-4270-8979-6a1eab5c0586" (UID: "cd1327d2-92f9-4270-8979-6a1eab5c0586"). InnerVolumeSpecName "kube-api-access-2fz26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:43:48.819217 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.819175    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd1327d2-92f9-4270-8979-6a1eab5c0586-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cd1327d2-92f9-4270-8979-6a1eab5c0586" (UID: "cd1327d2-92f9-4270-8979-6a1eab5c0586"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:43:48.917975 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.917940    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd1327d2-92f9-4270-8979-6a1eab5c0586-openshift-service-ca\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:43:48.917975 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.917968    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjg7l\" (UniqueName: \"kubernetes.io/projected/76bef79f-1e1b-4549-9f68-f984b295e10b-kube-api-access-qjg7l\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:43:48.917975 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.917978    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fz26\" (UniqueName: \"kubernetes.io/projected/cd1327d2-92f9-4270-8979-6a1eab5c0586-kube-api-access-2fz26\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:43:48.918178 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.917987    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd1327d2-92f9-4270-8979-6a1eab5c0586-metrics-certs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:43:48.918178 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.917996    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd1327d2-92f9-4270-8979-6a1eab5c0586-maas-api-tls\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:43:48.918178 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:48.918003    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bef79f-1e1b-4549-9f68-f984b295e10b-plugins-config-volume\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:43:49.431673 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:49.431632    2566 generic.go:356] "Generic (PLEG): container finished" podID="76bef79f-1e1b-4549-9f68-f984b295e10b" containerID="418d042d7bbe7bcbf9f2e47d1587b469e132177b4a491362ed20f9fbe81911b5" exitCode=0
Sep 06 19:43:49.431673 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:49.431671    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2"
Sep 06 19:43:49.432270 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:49.431700    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2" event={"ID":"76bef79f-1e1b-4549-9f68-f984b295e10b","Type":"ContainerDied","Data":"418d042d7bbe7bcbf9f2e47d1587b469e132177b4a491362ed20f9fbe81911b5"}
Sep 06 19:43:49.432270 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:49.431757    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2" event={"ID":"76bef79f-1e1b-4549-9f68-f984b295e10b","Type":"ContainerDied","Data":"7ebafd02555b6bd590c05c65d0911df1a57d68822fb560b26414123503119d42"}
Sep 06 19:43:49.432270 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:49.431784    2566 scope.go:117] "RemoveContainer" containerID="418d042d7bbe7bcbf9f2e47d1587b469e132177b4a491362ed20f9fbe81911b5"
Sep 06 19:43:49.433166 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:49.433139    2566 generic.go:356] "Generic (PLEG): container finished" podID="cd1327d2-92f9-4270-8979-6a1eab5c0586" containerID="1c730303cdda6dd63da44fbb91438c3a9e0dc46aba1b7b4bbd304f4fb3db4f63" exitCode=0
Sep 06 19:43:49.433324 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:49.433166    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m" event={"ID":"cd1327d2-92f9-4270-8979-6a1eab5c0586","Type":"ContainerDied","Data":"1c730303cdda6dd63da44fbb91438c3a9e0dc46aba1b7b4bbd304f4fb3db4f63"}
Sep 06 19:43:49.433324 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:49.433190    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"
Sep 06 19:43:49.433324 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:49.433218    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m" event={"ID":"cd1327d2-92f9-4270-8979-6a1eab5c0586","Type":"ContainerDied","Data":"0d5d3166a4b52868ed1411d3821324ad303b3934e18fc481b80d8f806f7ed69c"}
Sep 06 19:43:49.442139 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:49.442124    2566 scope.go:117] "RemoveContainer" containerID="418d042d7bbe7bcbf9f2e47d1587b469e132177b4a491362ed20f9fbe81911b5"
Sep 06 19:43:49.442438 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:43:49.442416    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"418d042d7bbe7bcbf9f2e47d1587b469e132177b4a491362ed20f9fbe81911b5\": container with ID starting with 418d042d7bbe7bcbf9f2e47d1587b469e132177b4a491362ed20f9fbe81911b5 not found: ID does not exist" containerID="418d042d7bbe7bcbf9f2e47d1587b469e132177b4a491362ed20f9fbe81911b5"
Sep 06 19:43:49.442502 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:49.442446    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"418d042d7bbe7bcbf9f2e47d1587b469e132177b4a491362ed20f9fbe81911b5"} err="failed to get container status \"418d042d7bbe7bcbf9f2e47d1587b469e132177b4a491362ed20f9fbe81911b5\": rpc error: code = NotFound desc = could not find container \"418d042d7bbe7bcbf9f2e47d1587b469e132177b4a491362ed20f9fbe81911b5\": container with ID starting with 418d042d7bbe7bcbf9f2e47d1587b469e132177b4a491362ed20f9fbe81911b5 not found: ID does not exist"
Sep 06 19:43:49.442502 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:49.442464    2566 scope.go:117] "RemoveContainer" containerID="1c730303cdda6dd63da44fbb91438c3a9e0dc46aba1b7b4bbd304f4fb3db4f63"
Sep 06 19:43:49.455776 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:49.455730    2566 scope.go:117] "RemoveContainer" containerID="1c730303cdda6dd63da44fbb91438c3a9e0dc46aba1b7b4bbd304f4fb3db4f63"
Sep 06 19:43:49.456013 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:43:49.455992    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c730303cdda6dd63da44fbb91438c3a9e0dc46aba1b7b4bbd304f4fb3db4f63\": container with ID starting with 1c730303cdda6dd63da44fbb91438c3a9e0dc46aba1b7b4bbd304f4fb3db4f63 not found: ID does not exist" containerID="1c730303cdda6dd63da44fbb91438c3a9e0dc46aba1b7b4bbd304f4fb3db4f63"
Sep 06 19:43:49.456083 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:49.456025    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c730303cdda6dd63da44fbb91438c3a9e0dc46aba1b7b4bbd304f4fb3db4f63"} err="failed to get container status \"1c730303cdda6dd63da44fbb91438c3a9e0dc46aba1b7b4bbd304f4fb3db4f63\": rpc error: code = NotFound desc = could not find container \"1c730303cdda6dd63da44fbb91438c3a9e0dc46aba1b7b4bbd304f4fb3db4f63\": container with ID starting with 1c730303cdda6dd63da44fbb91438c3a9e0dc46aba1b7b4bbd304f4fb3db4f63 not found: ID does not exist"
Sep 06 19:43:49.469201 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:49.469178    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"]
Sep 06 19:43:49.477259 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:49.477236    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-88ac22-655fd6f578-7tk5m"]
Sep 06 19:43:49.486799 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:49.486766    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2"]
Sep 06 19:43:49.488737 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:49.488718    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-88ac22-54794df56c-tg6b2"]
Sep 06 19:43:50.310467 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:50.310433    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76bef79f-1e1b-4549-9f68-f984b295e10b" path="/var/lib/kubelet/pods/76bef79f-1e1b-4549-9f68-f984b295e10b/volumes"
Sep 06 19:43:50.310854 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:50.310839    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd1327d2-92f9-4270-8979-6a1eab5c0586" path="/var/lib/kubelet/pods/cd1327d2-92f9-4270-8979-6a1eab5c0586/volumes"
Sep 06 19:43:52.451557 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:43:52.451517    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:44:05.826729 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:05.826648    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"]
Sep 06 19:44:05.827068 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:05.826864    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr" podUID="355f0e9b-89be-4768-ab1a-4d6ca4b56b72" containerName="maas-api" containerID="cri-o://f9c823f6c47b5d11ace029f1ea94410e3a4c838afda959102163e9172eb9376b" gracePeriod=30
Sep 06 19:44:05.889478 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:05.889445    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t"]
Sep 06 19:44:05.890067 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:05.889936    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t" podUID="0126ebcf-90d4-41e5-a692-d2543e6441d2" containerName="payload-processing" containerID="cri-o://35a517016ae406e4e4e972ebcb14ed4c4831053e2c7e7abe9ae8f34b7020cc4c" gracePeriod=30
Sep 06 19:44:05.894903 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:05.894877    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn"]
Sep 06 19:44:05.895239 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:05.895210    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn" podUID="a0a4b1ad-2580-49e5-b64f-a5080137aa8f" containerName="payload-pre-processing" containerID="cri-o://e1c594a3be33d035fd48ed83093788522e797edc7049bc01814e4331745e79eb" gracePeriod=30
Sep 06 19:44:06.169239 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.169208    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn"
Sep 06 19:44:06.172663 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.172646    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"
Sep 06 19:44:06.175826 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.175808    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t"
Sep 06 19:44:06.272193 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.272155    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0126ebcf-90d4-41e5-a692-d2543e6441d2-plugins-config-volume\") pod \"0126ebcf-90d4-41e5-a692-d2543e6441d2\" (UID: \"0126ebcf-90d4-41e5-a692-d2543e6441d2\") "
Sep 06 19:44:06.272395 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.272207    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptx2v\" (UniqueName: \"kubernetes.io/projected/0126ebcf-90d4-41e5-a692-d2543e6441d2-kube-api-access-ptx2v\") pod \"0126ebcf-90d4-41e5-a692-d2543e6441d2\" (UID: \"0126ebcf-90d4-41e5-a692-d2543e6441d2\") "
Sep 06 19:44:06.272395 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.272236    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0a4b1ad-2580-49e5-b64f-a5080137aa8f-plugins-config-volume\") pod \"a0a4b1ad-2580-49e5-b64f-a5080137aa8f\" (UID: \"a0a4b1ad-2580-49e5-b64f-a5080137aa8f\") "
Sep 06 19:44:06.272395 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.272316    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-metrics-certs\") pod \"355f0e9b-89be-4768-ab1a-4d6ca4b56b72\" (UID: \"355f0e9b-89be-4768-ab1a-4d6ca4b56b72\") "
Sep 06 19:44:06.272395 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.272347    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-openshift-service-ca\") pod \"355f0e9b-89be-4768-ab1a-4d6ca4b56b72\" (UID: \"355f0e9b-89be-4768-ab1a-4d6ca4b56b72\") "
Sep 06 19:44:06.272395 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.272388    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-maas-api-tls\") pod \"355f0e9b-89be-4768-ab1a-4d6ca4b56b72\" (UID: \"355f0e9b-89be-4768-ab1a-4d6ca4b56b72\") "
Sep 06 19:44:06.272684 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.272422    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9mg9\" (UniqueName: \"kubernetes.io/projected/a0a4b1ad-2580-49e5-b64f-a5080137aa8f-kube-api-access-s9mg9\") pod \"a0a4b1ad-2580-49e5-b64f-a5080137aa8f\" (UID: \"a0a4b1ad-2580-49e5-b64f-a5080137aa8f\") "
Sep 06 19:44:06.272684 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.272455    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwxsg\" (UniqueName: \"kubernetes.io/projected/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-kube-api-access-dwxsg\") pod \"355f0e9b-89be-4768-ab1a-4d6ca4b56b72\" (UID: \"355f0e9b-89be-4768-ab1a-4d6ca4b56b72\") "
Sep 06 19:44:06.272784 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.272736    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "355f0e9b-89be-4768-ab1a-4d6ca4b56b72" (UID: "355f0e9b-89be-4768-ab1a-4d6ca4b56b72"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:44:06.272784 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.272744    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a4b1ad-2580-49e5-b64f-a5080137aa8f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a0a4b1ad-2580-49e5-b64f-a5080137aa8f" (UID: "a0a4b1ad-2580-49e5-b64f-a5080137aa8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:44:06.272889 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.272854    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0126ebcf-90d4-41e5-a692-d2543e6441d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0126ebcf-90d4-41e5-a692-d2543e6441d2" (UID: "0126ebcf-90d4-41e5-a692-d2543e6441d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:44:06.274747 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.274717    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "355f0e9b-89be-4768-ab1a-4d6ca4b56b72" (UID: "355f0e9b-89be-4768-ab1a-4d6ca4b56b72"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:44:06.274861 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.274768    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a4b1ad-2580-49e5-b64f-a5080137aa8f-kube-api-access-s9mg9" (OuterVolumeSpecName: "kube-api-access-s9mg9") pod "a0a4b1ad-2580-49e5-b64f-a5080137aa8f" (UID: "a0a4b1ad-2580-49e5-b64f-a5080137aa8f"). InnerVolumeSpecName "kube-api-access-s9mg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:44:06.275004 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.274986    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "355f0e9b-89be-4768-ab1a-4d6ca4b56b72" (UID: "355f0e9b-89be-4768-ab1a-4d6ca4b56b72"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:44:06.275046 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.275012    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0126ebcf-90d4-41e5-a692-d2543e6441d2-kube-api-access-ptx2v" (OuterVolumeSpecName: "kube-api-access-ptx2v") pod "0126ebcf-90d4-41e5-a692-d2543e6441d2" (UID: "0126ebcf-90d4-41e5-a692-d2543e6441d2"). InnerVolumeSpecName "kube-api-access-ptx2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:44:06.275152 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.275138    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-kube-api-access-dwxsg" (OuterVolumeSpecName: "kube-api-access-dwxsg") pod "355f0e9b-89be-4768-ab1a-4d6ca4b56b72" (UID: "355f0e9b-89be-4768-ab1a-4d6ca4b56b72"). InnerVolumeSpecName "kube-api-access-dwxsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:44:06.373696 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.373625    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-metrics-certs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:44:06.373696 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.373652    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-openshift-service-ca\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:44:06.373696 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.373666    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-maas-api-tls\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:44:06.373696 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.373678    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9mg9\" (UniqueName: \"kubernetes.io/projected/a0a4b1ad-2580-49e5-b64f-a5080137aa8f-kube-api-access-s9mg9\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:44:06.373696 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.373689    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwxsg\" (UniqueName: \"kubernetes.io/projected/355f0e9b-89be-4768-ab1a-4d6ca4b56b72-kube-api-access-dwxsg\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:44:06.373925 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.373704    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0126ebcf-90d4-41e5-a692-d2543e6441d2-plugins-config-volume\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:44:06.373925 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.373716    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptx2v\" (UniqueName: \"kubernetes.io/projected/0126ebcf-90d4-41e5-a692-d2543e6441d2-kube-api-access-ptx2v\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:44:06.373925 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.373730    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0a4b1ad-2580-49e5-b64f-a5080137aa8f-plugins-config-volume\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:44:06.532859 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.532826    2566 generic.go:356] "Generic (PLEG): container finished" podID="a0a4b1ad-2580-49e5-b64f-a5080137aa8f" containerID="e1c594a3be33d035fd48ed83093788522e797edc7049bc01814e4331745e79eb" exitCode=0
Sep 06 19:44:06.532994 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.532879    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn"
Sep 06 19:44:06.532994 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.532910    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn" event={"ID":"a0a4b1ad-2580-49e5-b64f-a5080137aa8f","Type":"ContainerDied","Data":"e1c594a3be33d035fd48ed83093788522e797edc7049bc01814e4331745e79eb"}
Sep 06 19:44:06.532994 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.532961    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn" event={"ID":"a0a4b1ad-2580-49e5-b64f-a5080137aa8f","Type":"ContainerDied","Data":"fcb8bb6735e0514e787a0f32e4bc9dd010f01c144edf21018102c703c4d7f66c"}
Sep 06 19:44:06.532994 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.532985    2566 scope.go:117] "RemoveContainer" containerID="e1c594a3be33d035fd48ed83093788522e797edc7049bc01814e4331745e79eb"
Sep 06 19:44:06.534579 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.534556    2566 generic.go:356] "Generic (PLEG): container finished" podID="0126ebcf-90d4-41e5-a692-d2543e6441d2" containerID="35a517016ae406e4e4e972ebcb14ed4c4831053e2c7e7abe9ae8f34b7020cc4c" exitCode=0
Sep 06 19:44:06.534814 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.534671    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t"
Sep 06 19:44:06.534942 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.534690    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t" event={"ID":"0126ebcf-90d4-41e5-a692-d2543e6441d2","Type":"ContainerDied","Data":"35a517016ae406e4e4e972ebcb14ed4c4831053e2c7e7abe9ae8f34b7020cc4c"}
Sep 06 19:44:06.535042 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.534954    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t" event={"ID":"0126ebcf-90d4-41e5-a692-d2543e6441d2","Type":"ContainerDied","Data":"5c63488b095218f5d61aa956f4b325f33f66cdaaf116b28d00e56f10b0ecb591"}
Sep 06 19:44:06.536398 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.536376    2566 generic.go:356] "Generic (PLEG): container finished" podID="355f0e9b-89be-4768-ab1a-4d6ca4b56b72" containerID="f9c823f6c47b5d11ace029f1ea94410e3a4c838afda959102163e9172eb9376b" exitCode=0
Sep 06 19:44:06.536489 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.536412    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr" event={"ID":"355f0e9b-89be-4768-ab1a-4d6ca4b56b72","Type":"ContainerDied","Data":"f9c823f6c47b5d11ace029f1ea94410e3a4c838afda959102163e9172eb9376b"}
Sep 06 19:44:06.536489 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.536419    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"
Sep 06 19:44:06.536489 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.536448    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr" event={"ID":"355f0e9b-89be-4768-ab1a-4d6ca4b56b72","Type":"ContainerDied","Data":"4c1848257481279e8db8a60425c094a88bd2e81f11bd0678f6ee7c7f72910ca8"}
Sep 06 19:44:06.543230 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.543215    2566 scope.go:117] "RemoveContainer" containerID="e1c594a3be33d035fd48ed83093788522e797edc7049bc01814e4331745e79eb"
Sep 06 19:44:06.543533 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:44:06.543514    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c594a3be33d035fd48ed83093788522e797edc7049bc01814e4331745e79eb\": container with ID starting with e1c594a3be33d035fd48ed83093788522e797edc7049bc01814e4331745e79eb not found: ID does not exist" containerID="e1c594a3be33d035fd48ed83093788522e797edc7049bc01814e4331745e79eb"
Sep 06 19:44:06.543591 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.543542    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c594a3be33d035fd48ed83093788522e797edc7049bc01814e4331745e79eb"} err="failed to get container status \"e1c594a3be33d035fd48ed83093788522e797edc7049bc01814e4331745e79eb\": rpc error: code = NotFound desc = could not find container \"e1c594a3be33d035fd48ed83093788522e797edc7049bc01814e4331745e79eb\": container with ID starting with e1c594a3be33d035fd48ed83093788522e797edc7049bc01814e4331745e79eb not found: ID does not exist"
Sep 06 19:44:06.543591 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.543558    2566 scope.go:117] "RemoveContainer" containerID="35a517016ae406e4e4e972ebcb14ed4c4831053e2c7e7abe9ae8f34b7020cc4c"
Sep 06 19:44:06.551858 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.551840    2566 scope.go:117] "RemoveContainer" containerID="35a517016ae406e4e4e972ebcb14ed4c4831053e2c7e7abe9ae8f34b7020cc4c"
Sep 06 19:44:06.552125 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:44:06.552107    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a517016ae406e4e4e972ebcb14ed4c4831053e2c7e7abe9ae8f34b7020cc4c\": container with ID starting with 35a517016ae406e4e4e972ebcb14ed4c4831053e2c7e7abe9ae8f34b7020cc4c not found: ID does not exist" containerID="35a517016ae406e4e4e972ebcb14ed4c4831053e2c7e7abe9ae8f34b7020cc4c"
Sep 06 19:44:06.552183 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.552131    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a517016ae406e4e4e972ebcb14ed4c4831053e2c7e7abe9ae8f34b7020cc4c"} err="failed to get container status \"35a517016ae406e4e4e972ebcb14ed4c4831053e2c7e7abe9ae8f34b7020cc4c\": rpc error: code = NotFound desc = could not find container \"35a517016ae406e4e4e972ebcb14ed4c4831053e2c7e7abe9ae8f34b7020cc4c\": container with ID starting with 35a517016ae406e4e4e972ebcb14ed4c4831053e2c7e7abe9ae8f34b7020cc4c not found: ID does not exist"
Sep 06 19:44:06.552183 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.552145    2566 scope.go:117] "RemoveContainer" containerID="f9c823f6c47b5d11ace029f1ea94410e3a4c838afda959102163e9172eb9376b"
Sep 06 19:44:06.556595 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.556578    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn"]
Sep 06 19:44:06.560129 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.560109    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e5ef25-dbf45b586-mppzn"]
Sep 06 19:44:06.560724 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.560705    2566 scope.go:117] "RemoveContainer" containerID="f9c823f6c47b5d11ace029f1ea94410e3a4c838afda959102163e9172eb9376b"
Sep 06 19:44:06.560956 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:44:06.560937    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9c823f6c47b5d11ace029f1ea94410e3a4c838afda959102163e9172eb9376b\": container with ID starting with f9c823f6c47b5d11ace029f1ea94410e3a4c838afda959102163e9172eb9376b not found: ID does not exist" containerID="f9c823f6c47b5d11ace029f1ea94410e3a4c838afda959102163e9172eb9376b"
Sep 06 19:44:06.561019 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.560962    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c823f6c47b5d11ace029f1ea94410e3a4c838afda959102163e9172eb9376b"} err="failed to get container status \"f9c823f6c47b5d11ace029f1ea94410e3a4c838afda959102163e9172eb9376b\": rpc error: code = NotFound desc = could not find container \"f9c823f6c47b5d11ace029f1ea94410e3a4c838afda959102163e9172eb9376b\": container with ID starting with f9c823f6c47b5d11ace029f1ea94410e3a4c838afda959102163e9172eb9376b not found: ID does not exist"
Sep 06 19:44:06.570364 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.570344    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"]
Sep 06 19:44:06.572273 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.572254    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e5ef25-78f6845b66-tb8kr"]
Sep 06 19:44:06.581931 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.581913    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t"]
Sep 06 19:44:06.585436 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.585419    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e5ef25-8659bbb8bc-kvk8t"]
Sep 06 19:44:06.722045 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:06.722019    2566 scope.go:117] "RemoveContainer" containerID="89cf465bd1e280bd3711ccc09c490ecdfa7c477fe95ff3dfd686b2f91225699e"
Sep 06 19:44:08.310625 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:08.310574    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0126ebcf-90d4-41e5-a692-d2543e6441d2" path="/var/lib/kubelet/pods/0126ebcf-90d4-41e5-a692-d2543e6441d2/volumes"
Sep 06 19:44:08.310998 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:08.310976    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="355f0e9b-89be-4768-ab1a-4d6ca4b56b72" path="/var/lib/kubelet/pods/355f0e9b-89be-4768-ab1a-4d6ca4b56b72/volumes"
Sep 06 19:44:08.311283 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:08.311269    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0a4b1ad-2580-49e5-b64f-a5080137aa8f" path="/var/lib/kubelet/pods/a0a4b1ad-2580-49e5-b64f-a5080137aa8f/volumes"
Sep 06 19:44:08.550355 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:08.550315    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:44:23.428384 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:23.428347    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"]
Sep 06 19:44:23.428910 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:23.428624    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h" podUID="40fdea58-341d-4f7c-9815-60776ddf99a7" containerName="maas-api" containerID="cri-o://a18a8b7ee5b51e020febee0db8a095e1312904dfd8e62518f213ecbe96f11969" gracePeriod=30
Sep 06 19:44:23.631264 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:23.631233    2566 generic.go:356] "Generic (PLEG): container finished" podID="40fdea58-341d-4f7c-9815-60776ddf99a7" containerID="a18a8b7ee5b51e020febee0db8a095e1312904dfd8e62518f213ecbe96f11969" exitCode=0
Sep 06 19:44:23.631427 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:23.631304    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h" event={"ID":"40fdea58-341d-4f7c-9815-60776ddf99a7","Type":"ContainerDied","Data":"a18a8b7ee5b51e020febee0db8a095e1312904dfd8e62518f213ecbe96f11969"}
Sep 06 19:44:23.674985 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:23.674956    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"
Sep 06 19:44:23.831870 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:23.831839    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40fdea58-341d-4f7c-9815-60776ddf99a7-openshift-service-ca\") pod \"40fdea58-341d-4f7c-9815-60776ddf99a7\" (UID: \"40fdea58-341d-4f7c-9815-60776ddf99a7\") "
Sep 06 19:44:23.832056 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:23.831907    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcrf5\" (UniqueName: \"kubernetes.io/projected/40fdea58-341d-4f7c-9815-60776ddf99a7-kube-api-access-bcrf5\") pod \"40fdea58-341d-4f7c-9815-60776ddf99a7\" (UID: \"40fdea58-341d-4f7c-9815-60776ddf99a7\") "
Sep 06 19:44:23.832056 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:23.831941    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40fdea58-341d-4f7c-9815-60776ddf99a7-maas-api-tls\") pod \"40fdea58-341d-4f7c-9815-60776ddf99a7\" (UID: \"40fdea58-341d-4f7c-9815-60776ddf99a7\") "
Sep 06 19:44:23.832056 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:23.831977    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40fdea58-341d-4f7c-9815-60776ddf99a7-metrics-certs\") pod \"40fdea58-341d-4f7c-9815-60776ddf99a7\" (UID: \"40fdea58-341d-4f7c-9815-60776ddf99a7\") "
Sep 06 19:44:23.832291 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:23.832264    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40fdea58-341d-4f7c-9815-60776ddf99a7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "40fdea58-341d-4f7c-9815-60776ddf99a7" (UID: "40fdea58-341d-4f7c-9815-60776ddf99a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:44:23.834210 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:23.834185    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40fdea58-341d-4f7c-9815-60776ddf99a7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "40fdea58-341d-4f7c-9815-60776ddf99a7" (UID: "40fdea58-341d-4f7c-9815-60776ddf99a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:44:23.834333 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:23.834228    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40fdea58-341d-4f7c-9815-60776ddf99a7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "40fdea58-341d-4f7c-9815-60776ddf99a7" (UID: "40fdea58-341d-4f7c-9815-60776ddf99a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:44:23.834333 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:23.834241    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40fdea58-341d-4f7c-9815-60776ddf99a7-kube-api-access-bcrf5" (OuterVolumeSpecName: "kube-api-access-bcrf5") pod "40fdea58-341d-4f7c-9815-60776ddf99a7" (UID: "40fdea58-341d-4f7c-9815-60776ddf99a7"). InnerVolumeSpecName "kube-api-access-bcrf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:44:23.932906 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:23.932879    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcrf5\" (UniqueName: \"kubernetes.io/projected/40fdea58-341d-4f7c-9815-60776ddf99a7-kube-api-access-bcrf5\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:44:23.932906 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:23.932901    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40fdea58-341d-4f7c-9815-60776ddf99a7-maas-api-tls\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:44:23.932906 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:23.932911    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40fdea58-341d-4f7c-9815-60776ddf99a7-metrics-certs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:44:23.933096 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:23.932918    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40fdea58-341d-4f7c-9815-60776ddf99a7-openshift-service-ca\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:44:24.638115 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:24.638039    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"
Sep 06 19:44:24.638525 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:24.638039    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h" event={"ID":"40fdea58-341d-4f7c-9815-60776ddf99a7","Type":"ContainerDied","Data":"d8d1b00152d260c7e5262e37afd13a0e6d1e8444f1d9f83bf5f3368fbec8ef36"}
Sep 06 19:44:24.638525 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:24.638167    2566 scope.go:117] "RemoveContainer" containerID="a18a8b7ee5b51e020febee0db8a095e1312904dfd8e62518f213ecbe96f11969"
Sep 06 19:44:24.654075 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:24.654053    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"]
Sep 06 19:44:24.657444 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:24.657415    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-476f98-8964c4dd9-wr42h"]
Sep 06 19:44:25.949418 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:25.949388    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:44:26.310849 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:26.310768    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40fdea58-341d-4f7c-9815-60776ddf99a7" path="/var/lib/kubelet/pods/40fdea58-341d-4f7c-9815-60776ddf99a7/volumes"
Sep 06 19:44:33.849196 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:44:33.849157    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:45:06.765178 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:45:06.765144    2566 scope.go:117] "RemoveContainer" containerID="4bc8109950316ba7f21b3551d6f890a44d3d109056c6fd6b43264ebec9e95978"
Sep 06 19:45:56.942422 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:45:56.942382    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:46:06.505799 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:46:06.505770    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnssq_9254e6f1-b73d-49cc-8bc1-513925ccde4f/ovn-acl-logging/0.log"
Sep 06 19:46:06.512481 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:46:06.512461    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnssq_9254e6f1-b73d-49cc-8bc1-513925ccde4f/ovn-acl-logging/0.log"
Sep 06 19:46:23.446785 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:46:23.446749    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:46:31.446082 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:46:31.446053    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:46:38.538997 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:46:38.538957    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:46:47.042414 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:46:47.042332    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:46:59.544461 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:46:59.544363    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:47:07.546983 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:47:07.546949    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:48:35.048480 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:48:35.048396    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:48:42.951620 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:48:42.951582    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:48:49.044166 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:48:49.044126    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:49:08.740790 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.740755    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"]
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741082    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="355f0e9b-89be-4768-ab1a-4d6ca4b56b72" containerName="maas-api"
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741093    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="355f0e9b-89be-4768-ab1a-4d6ca4b56b72" containerName="maas-api"
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741105    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76bef79f-1e1b-4549-9f68-f984b295e10b" containerName="payload-processing"
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741111    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="76bef79f-1e1b-4549-9f68-f984b295e10b" containerName="payload-processing"
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741121    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd1327d2-92f9-4270-8979-6a1eab5c0586" containerName="maas-api"
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741126    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd1327d2-92f9-4270-8979-6a1eab5c0586" containerName="maas-api"
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741136    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0a4b1ad-2580-49e5-b64f-a5080137aa8f" containerName="payload-pre-processing"
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741142    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0a4b1ad-2580-49e5-b64f-a5080137aa8f" containerName="payload-pre-processing"
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741151    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06e05ac3-0923-4d7a-a2e3-7c426db583f2" containerName="payload-pre-processing"
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741157    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="06e05ac3-0923-4d7a-a2e3-7c426db583f2" containerName="payload-pre-processing"
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741169    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0126ebcf-90d4-41e5-a692-d2543e6441d2" containerName="payload-processing"
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741175    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0126ebcf-90d4-41e5-a692-d2543e6441d2" containerName="payload-processing"
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741180    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40fdea58-341d-4f7c-9815-60776ddf99a7" containerName="maas-api"
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741185    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="40fdea58-341d-4f7c-9815-60776ddf99a7" containerName="maas-api"
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741190    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c9aeeaf-9e6a-4a8f-8605-131329bda74d" containerName="main"
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741194    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c9aeeaf-9e6a-4a8f-8605-131329bda74d" containerName="main"
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741259    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="40fdea58-341d-4f7c-9815-60776ddf99a7" containerName="maas-api"
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741267    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0126ebcf-90d4-41e5-a692-d2543e6441d2" containerName="payload-processing"
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741273    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="76bef79f-1e1b-4549-9f68-f984b295e10b" containerName="payload-processing"
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741280    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd1327d2-92f9-4270-8979-6a1eab5c0586" containerName="maas-api"
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741285    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="06e05ac3-0923-4d7a-a2e3-7c426db583f2" containerName="payload-pre-processing"
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741292    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0a4b1ad-2580-49e5-b64f-a5080137aa8f" containerName="payload-pre-processing"
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741297    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="355f0e9b-89be-4768-ab1a-4d6ca4b56b72" containerName="maas-api"
Sep 06 19:49:08.741290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.741302    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c9aeeaf-9e6a-4a8f-8605-131329bda74d" containerName="main"
Sep 06 19:49:08.744335 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.744319    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:08.746688 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.746673    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-8656aba7-kserve-self-signed-certs\""
Sep 06 19:49:08.752592 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.752570    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"]
Sep 06 19:49:08.839948 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.839917    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-home\") pod \"e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") " pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:08.839948 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.839958    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-model-cache\") pod \"e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") " pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:08.840128 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.840025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-tls-certs\") pod \"e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") " pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:08.840128 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.840059    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-dshm\") pod \"e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") " pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:08.840128 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.840093    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-tmp-dir\") pod \"e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") " pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:08.840128 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.840110    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkmjd\" (UniqueName: \"kubernetes.io/projected/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-kube-api-access-lkmjd\") pod \"e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") " pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:08.941122 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.941092    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-tmp-dir\") pod \"e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") " pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:08.941122 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.941122    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkmjd\" (UniqueName: \"kubernetes.io/projected/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-kube-api-access-lkmjd\") pod \"e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") " pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:08.941307 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.941156    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-home\") pod \"e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") " pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:08.941307 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.941181    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-model-cache\") pod \"e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") " pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:08.941307 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.941235    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-tls-certs\") pod \"e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") " pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:08.941307 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.941266    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-dshm\") pod \"e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") " pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:08.941544 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.941524    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-tmp-dir\") pod \"e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") " pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:08.941632 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.941585    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-home\") pod \"e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") " pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:08.941696 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.941598    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-model-cache\") pod \"e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") " pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:08.943705 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.943680    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-dshm\") pod \"e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") " pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:08.943949 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.943931    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-tls-certs\") pod \"e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") " pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:08.948784 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:08.948765    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkmjd\" (UniqueName: \"kubernetes.io/projected/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-kube-api-access-lkmjd\") pod \"e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") " pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:09.054063 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:09.053994    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:09.325025 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:09.324994    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"]
Sep 06 19:49:09.326487 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:49:09.326459    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6e9a2ec_89f8_476a_89ab_8eef14ac657c.slice/crio-4b10506baf3404b16e5193e94acb2f6f88013b686148d34ece650c573b6293b8 WatchSource:0}: Error finding container 4b10506baf3404b16e5193e94acb2f6f88013b686148d34ece650c573b6293b8: Status 404 returned error can't find the container with id 4b10506baf3404b16e5193e94acb2f6f88013b686148d34ece650c573b6293b8
Sep 06 19:49:10.139986 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:10.139943    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj" event={"ID":"f6e9a2ec-89f8-476a-89ab-8eef14ac657c","Type":"ContainerStarted","Data":"ca3b4490b8af5763b8f7c074482600704204cd7946779f09c033b4e0965fd4b4"}
Sep 06 19:49:10.139986 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:10.139989    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj" event={"ID":"f6e9a2ec-89f8-476a-89ab-8eef14ac657c","Type":"ContainerStarted","Data":"4b10506baf3404b16e5193e94acb2f6f88013b686148d34ece650c573b6293b8"}
Sep 06 19:49:10.155890 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:10.155841    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj" podStartSLOduration=2.155820067 podStartE2EDuration="2.155820067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:49:10.15481517 +0000 UTC m=+2284.335996975" watchObservedRunningTime="2026-09-06 19:49:10.155820067 +0000 UTC m=+2284.337001872"
Sep 06 19:49:11.053843 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:11.053807    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:49:19.054799 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:19.054760    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:19.054799 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:19.054810    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:19.067495 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:19.067464    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:19.200209 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:19.200184    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:49:45.148207 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.148175    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"]
Sep 06 19:49:45.151296 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.151279    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:49:45.153864 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.153841    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-8656aba7-kserve-self-signed-certs\""
Sep 06 19:49:45.165952 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.165928    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"]
Sep 06 19:49:45.260682 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.260652    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8826bd1a-98db-4e2c-9b26-49f4f7067772-tls-certs\") pod \"e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") " pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:49:45.260682 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.260684    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-dshm\") pod \"e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") " pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:49:45.260885 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.260707    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-home\") pod \"e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") " pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:49:45.260885 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.260822    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2psg\" (UniqueName: \"kubernetes.io/projected/8826bd1a-98db-4e2c-9b26-49f4f7067772-kube-api-access-k2psg\") pod \"e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") " pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:49:45.260994 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.260903    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-tmp-dir\") pod \"e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") " pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:49:45.260994 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.260960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-model-cache\") pod \"e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") " pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:49:45.361519 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.361485    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-home\") pod \"e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") " pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:49:45.361688 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.361538    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2psg\" (UniqueName: \"kubernetes.io/projected/8826bd1a-98db-4e2c-9b26-49f4f7067772-kube-api-access-k2psg\") pod \"e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") " pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:49:45.361688 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.361583    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-tmp-dir\") pod \"e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") " pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:49:45.361688 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.361669    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-model-cache\") pod \"e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") " pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:49:45.361806 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.361704    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8826bd1a-98db-4e2c-9b26-49f4f7067772-tls-certs\") pod \"e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") " pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:49:45.361806 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.361727    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-dshm\") pod \"e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") " pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:49:45.362003 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.361985    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-home\") pod \"e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") " pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:49:45.362040 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.362017    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-model-cache\") pod \"e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") " pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:49:45.362097 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.362074    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-tmp-dir\") pod \"e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") " pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:49:45.364131 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.364104    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-dshm\") pod \"e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") " pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:49:45.364299 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.364282    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8826bd1a-98db-4e2c-9b26-49f4f7067772-tls-certs\") pod \"e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") " pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:49:45.369213 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.369188    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2psg\" (UniqueName: \"kubernetes.io/projected/8826bd1a-98db-4e2c-9b26-49f4f7067772-kube-api-access-k2psg\") pod \"e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") " pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:49:45.466666 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.466643    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:49:45.738269 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:45.738243    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"]
Sep 06 19:49:45.740214 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:49:45.740184    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8826bd1a_98db_4e2c_9b26_49f4f7067772.slice/crio-7bedaff8a2826c9290a9a70fa535b4ef18e4ceb27ec713c8b6fdf8dce4def1f8 WatchSource:0}: Error finding container 7bedaff8a2826c9290a9a70fa535b4ef18e4ceb27ec713c8b6fdf8dce4def1f8: Status 404 returned error can't find the container with id 7bedaff8a2826c9290a9a70fa535b4ef18e4ceb27ec713c8b6fdf8dce4def1f8
Sep 06 19:49:46.333822 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:46.333784    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb" event={"ID":"8826bd1a-98db-4e2c-9b26-49f4f7067772","Type":"ContainerStarted","Data":"26ecbc14f02b574d70d6c7f68874e6acf7365ed1c7b33d89ea3c410a9c137c63"}
Sep 06 19:49:46.333822 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:46.333828    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb" event={"ID":"8826bd1a-98db-4e2c-9b26-49f4f7067772","Type":"ContainerStarted","Data":"7bedaff8a2826c9290a9a70fa535b4ef18e4ceb27ec713c8b6fdf8dce4def1f8"}
Sep 06 19:49:46.351403 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:46.351356    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb" podStartSLOduration=1.351343744 podStartE2EDuration="1.351343744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:49:46.348423618 +0000 UTC m=+2320.529605423" watchObservedRunningTime="2026-09-06 19:49:46.351343744 +0000 UTC m=+2320.532525560"
Sep 06 19:49:47.748256 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:47.748221    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:49:55.467045 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:55.467006    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:49:55.467438 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:55.467057    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:49:55.479409 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:55.479387    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:49:56.401883 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:49:56.401857    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:50:08.945156 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:08.945075    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"]
Sep 06 19:50:10.219070 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:10.219032    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"]
Sep 06 19:50:10.850921 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:10.850888    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:50:19.541079 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:19.541047    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:50:23.965711 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:23.965647    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj" podUID="f6e9a2ec-89f8-476a-89ab-8eef14ac657c" containerName="main" containerID="cri-o://ca3b4490b8af5763b8f7c074482600704204cd7946779f09c033b4e0965fd4b4" gracePeriod=45
Sep 06 19:50:24.212704 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.212682    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:50:24.312753 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.312688    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkmjd\" (UniqueName: \"kubernetes.io/projected/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-kube-api-access-lkmjd\") pod \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") "
Sep 06 19:50:24.312753 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.312747    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-home\") pod \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") "
Sep 06 19:50:24.312960 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.312763    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-tmp-dir\") pod \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") "
Sep 06 19:50:24.312960 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.312781    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-dshm\") pod \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") "
Sep 06 19:50:24.312960 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.312809    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-tls-certs\") pod \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") "
Sep 06 19:50:24.312960 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.312838    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-model-cache\") pod \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\" (UID: \"f6e9a2ec-89f8-476a-89ab-8eef14ac657c\") "
Sep 06 19:50:24.313168 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.312965    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-home" (OuterVolumeSpecName: "home") pod "f6e9a2ec-89f8-476a-89ab-8eef14ac657c" (UID: "f6e9a2ec-89f8-476a-89ab-8eef14ac657c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:50:24.313168 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.313069    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6e9a2ec-89f8-476a-89ab-8eef14ac657c" (UID: "f6e9a2ec-89f8-476a-89ab-8eef14ac657c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:50:24.313168 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.313079    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-home\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:50:24.313168 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.313136    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6e9a2ec-89f8-476a-89ab-8eef14ac657c" (UID: "f6e9a2ec-89f8-476a-89ab-8eef14ac657c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:50:24.314940 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.314913    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-dshm" (OuterVolumeSpecName: "dshm") pod "f6e9a2ec-89f8-476a-89ab-8eef14ac657c" (UID: "f6e9a2ec-89f8-476a-89ab-8eef14ac657c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:50:24.315044 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.314946    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6e9a2ec-89f8-476a-89ab-8eef14ac657c" (UID: "f6e9a2ec-89f8-476a-89ab-8eef14ac657c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:50:24.315044 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.315002    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-kube-api-access-lkmjd" (OuterVolumeSpecName: "kube-api-access-lkmjd") pod "f6e9a2ec-89f8-476a-89ab-8eef14ac657c" (UID: "f6e9a2ec-89f8-476a-89ab-8eef14ac657c"). InnerVolumeSpecName "kube-api-access-lkmjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:50:24.414170 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.414135    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-tls-certs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:50:24.414170 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.414165    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-model-cache\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:50:24.414320 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.414177    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkmjd\" (UniqueName: \"kubernetes.io/projected/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-kube-api-access-lkmjd\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:50:24.414320 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.414194    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-tmp-dir\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:50:24.414320 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.414208    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6e9a2ec-89f8-476a-89ab-8eef14ac657c-dshm\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:50:24.540277 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.540249    2566 generic.go:356] "Generic (PLEG): container finished" podID="f6e9a2ec-89f8-476a-89ab-8eef14ac657c" containerID="ca3b4490b8af5763b8f7c074482600704204cd7946779f09c033b4e0965fd4b4" exitCode=0
Sep 06 19:50:24.540387 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.540294    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"
Sep 06 19:50:24.540387 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.540300    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj" event={"ID":"f6e9a2ec-89f8-476a-89ab-8eef14ac657c","Type":"ContainerDied","Data":"ca3b4490b8af5763b8f7c074482600704204cd7946779f09c033b4e0965fd4b4"}
Sep 06 19:50:24.540387 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.540326    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj" event={"ID":"f6e9a2ec-89f8-476a-89ab-8eef14ac657c","Type":"ContainerDied","Data":"4b10506baf3404b16e5193e94acb2f6f88013b686148d34ece650c573b6293b8"}
Sep 06 19:50:24.540387 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.540341    2566 scope.go:117] "RemoveContainer" containerID="ca3b4490b8af5763b8f7c074482600704204cd7946779f09c033b4e0965fd4b4"
Sep 06 19:50:24.550181 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.550167    2566 scope.go:117] "RemoveContainer" containerID="ca3b4490b8af5763b8f7c074482600704204cd7946779f09c033b4e0965fd4b4"
Sep 06 19:50:24.550457 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:50:24.550440    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca3b4490b8af5763b8f7c074482600704204cd7946779f09c033b4e0965fd4b4\": container with ID starting with ca3b4490b8af5763b8f7c074482600704204cd7946779f09c033b4e0965fd4b4 not found: ID does not exist" containerID="ca3b4490b8af5763b8f7c074482600704204cd7946779f09c033b4e0965fd4b4"
Sep 06 19:50:24.550506 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.550464    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca3b4490b8af5763b8f7c074482600704204cd7946779f09c033b4e0965fd4b4"} err="failed to get container status \"ca3b4490b8af5763b8f7c074482600704204cd7946779f09c033b4e0965fd4b4\": rpc error: code = NotFound desc = could not find container \"ca3b4490b8af5763b8f7c074482600704204cd7946779f09c033b4e0965fd4b4\": container with ID starting with ca3b4490b8af5763b8f7c074482600704204cd7946779f09c033b4e0965fd4b4 not found: ID does not exist"
Sep 06 19:50:24.562234 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.562208    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"]
Sep 06 19:50:24.565730 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:24.565681    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-8656aba7-kserve-7cfff6857f-7vpbj"]
Sep 06 19:50:25.239338 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.239277    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb" podUID="8826bd1a-98db-4e2c-9b26-49f4f7067772" containerName="main" containerID="cri-o://26ecbc14f02b574d70d6c7f68874e6acf7365ed1c7b33d89ea3c410a9c137c63" gracePeriod=45
Sep 06 19:50:25.487914 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.487887    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:50:25.546930 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.546866    2566 generic.go:356] "Generic (PLEG): container finished" podID="8826bd1a-98db-4e2c-9b26-49f4f7067772" containerID="26ecbc14f02b574d70d6c7f68874e6acf7365ed1c7b33d89ea3c410a9c137c63" exitCode=0
Sep 06 19:50:25.546930 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.546909    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"
Sep 06 19:50:25.546930 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.546908    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb" event={"ID":"8826bd1a-98db-4e2c-9b26-49f4f7067772","Type":"ContainerDied","Data":"26ecbc14f02b574d70d6c7f68874e6acf7365ed1c7b33d89ea3c410a9c137c63"}
Sep 06 19:50:25.547133 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.546954    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb" event={"ID":"8826bd1a-98db-4e2c-9b26-49f4f7067772","Type":"ContainerDied","Data":"7bedaff8a2826c9290a9a70fa535b4ef18e4ceb27ec713c8b6fdf8dce4def1f8"}
Sep 06 19:50:25.547133 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.546976    2566 scope.go:117] "RemoveContainer" containerID="26ecbc14f02b574d70d6c7f68874e6acf7365ed1c7b33d89ea3c410a9c137c63"
Sep 06 19:50:25.555557 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.555537    2566 scope.go:117] "RemoveContainer" containerID="26ecbc14f02b574d70d6c7f68874e6acf7365ed1c7b33d89ea3c410a9c137c63"
Sep 06 19:50:25.555823 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:50:25.555805    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26ecbc14f02b574d70d6c7f68874e6acf7365ed1c7b33d89ea3c410a9c137c63\": container with ID starting with 26ecbc14f02b574d70d6c7f68874e6acf7365ed1c7b33d89ea3c410a9c137c63 not found: ID does not exist" containerID="26ecbc14f02b574d70d6c7f68874e6acf7365ed1c7b33d89ea3c410a9c137c63"
Sep 06 19:50:25.555907 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.555835    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26ecbc14f02b574d70d6c7f68874e6acf7365ed1c7b33d89ea3c410a9c137c63"} err="failed to get container status \"26ecbc14f02b574d70d6c7f68874e6acf7365ed1c7b33d89ea3c410a9c137c63\": rpc error: code = NotFound desc = could not find container \"26ecbc14f02b574d70d6c7f68874e6acf7365ed1c7b33d89ea3c410a9c137c63\": container with ID starting with 26ecbc14f02b574d70d6c7f68874e6acf7365ed1c7b33d89ea3c410a9c137c63 not found: ID does not exist"
Sep 06 19:50:25.623969 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.623936    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-tmp-dir\") pod \"8826bd1a-98db-4e2c-9b26-49f4f7067772\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") "
Sep 06 19:50:25.624117 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.624024    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2psg\" (UniqueName: \"kubernetes.io/projected/8826bd1a-98db-4e2c-9b26-49f4f7067772-kube-api-access-k2psg\") pod \"8826bd1a-98db-4e2c-9b26-49f4f7067772\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") "
Sep 06 19:50:25.624117 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.624064    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-home\") pod \"8826bd1a-98db-4e2c-9b26-49f4f7067772\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") "
Sep 06 19:50:25.624117 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.624093    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-model-cache\") pod \"8826bd1a-98db-4e2c-9b26-49f4f7067772\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") "
Sep 06 19:50:25.624292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.624200    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8826bd1a-98db-4e2c-9b26-49f4f7067772" (UID: "8826bd1a-98db-4e2c-9b26-49f4f7067772"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:50:25.624292 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.624209    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-dshm\") pod \"8826bd1a-98db-4e2c-9b26-49f4f7067772\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") "
Sep 06 19:50:25.624395 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.624307    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8826bd1a-98db-4e2c-9b26-49f4f7067772-tls-certs\") pod \"8826bd1a-98db-4e2c-9b26-49f4f7067772\" (UID: \"8826bd1a-98db-4e2c-9b26-49f4f7067772\") "
Sep 06 19:50:25.624395 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.624318    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-home" (OuterVolumeSpecName: "home") pod "8826bd1a-98db-4e2c-9b26-49f4f7067772" (UID: "8826bd1a-98db-4e2c-9b26-49f4f7067772"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:50:25.624395 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.624350    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-model-cache" (OuterVolumeSpecName: "model-cache") pod "8826bd1a-98db-4e2c-9b26-49f4f7067772" (UID: "8826bd1a-98db-4e2c-9b26-49f4f7067772"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:50:25.624650 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.624634    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-model-cache\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:50:25.624717 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.624655    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-tmp-dir\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:50:25.624717 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.624667    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-home\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:50:25.626310 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.626289    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8826bd1a-98db-4e2c-9b26-49f4f7067772-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8826bd1a-98db-4e2c-9b26-49f4f7067772" (UID: "8826bd1a-98db-4e2c-9b26-49f4f7067772"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:50:25.626310 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.626295    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8826bd1a-98db-4e2c-9b26-49f4f7067772-kube-api-access-k2psg" (OuterVolumeSpecName: "kube-api-access-k2psg") pod "8826bd1a-98db-4e2c-9b26-49f4f7067772" (UID: "8826bd1a-98db-4e2c-9b26-49f4f7067772"). InnerVolumeSpecName "kube-api-access-k2psg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:50:25.626420 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.626290    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-dshm" (OuterVolumeSpecName: "dshm") pod "8826bd1a-98db-4e2c-9b26-49f4f7067772" (UID: "8826bd1a-98db-4e2c-9b26-49f4f7067772"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:50:25.725269 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.725241    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8826bd1a-98db-4e2c-9b26-49f4f7067772-dshm\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:50:25.725269 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.725265    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8826bd1a-98db-4e2c-9b26-49f4f7067772-tls-certs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:50:25.725405 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.725276    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2psg\" (UniqueName: \"kubernetes.io/projected/8826bd1a-98db-4e2c-9b26-49f4f7067772-kube-api-access-k2psg\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:50:25.867855 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.867822    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"]
Sep 06 19:50:25.871100 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:25.871077    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-8656aba7-kserve-544fd7c5c5-8m7jb"]
Sep 06 19:50:26.316419 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:26.316385    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8826bd1a-98db-4e2c-9b26-49f4f7067772" path="/var/lib/kubelet/pods/8826bd1a-98db-4e2c-9b26-49f4f7067772/volumes"
Sep 06 19:50:26.316795 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:26.316774    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6e9a2ec-89f8-476a-89ab-8eef14ac657c" path="/var/lib/kubelet/pods/f6e9a2ec-89f8-476a-89ab-8eef14ac657c/volumes"
Sep 06 19:50:26.643559 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:26.643480    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:50:32.951520 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:32.951486    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:50:33.280033 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.279959    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"]
Sep 06 19:50:33.280313 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.280302    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e9a2ec-89f8-476a-89ab-8eef14ac657c" containerName="main"
Sep 06 19:50:33.280353 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.280315    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e9a2ec-89f8-476a-89ab-8eef14ac657c" containerName="main"
Sep 06 19:50:33.280353 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.280332    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8826bd1a-98db-4e2c-9b26-49f4f7067772" containerName="main"
Sep 06 19:50:33.280353 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.280337    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8826bd1a-98db-4e2c-9b26-49f4f7067772" containerName="main"
Sep 06 19:50:33.280446 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.280393    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8826bd1a-98db-4e2c-9b26-49f4f7067772" containerName="main"
Sep 06 19:50:33.280446 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.280401    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6e9a2ec-89f8-476a-89ab-8eef14ac657c" containerName="main"
Sep 06 19:50:33.284759 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.284740    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"
Sep 06 19:50:33.286000 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.285976    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-992dv\" (UniqueName: \"kubernetes.io/projected/b251c64b-ea33-428c-8664-1c80333cace9-kube-api-access-992dv\") pod \"maas-api-5cb9864d77-trrxz\" (UID: \"b251c64b-ea33-428c-8664-1c80333cace9\") " pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"
Sep 06 19:50:33.286112 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.286030    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b251c64b-ea33-428c-8664-1c80333cace9-maas-api-tls\") pod \"maas-api-5cb9864d77-trrxz\" (UID: \"b251c64b-ea33-428c-8664-1c80333cace9\") " pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"
Sep 06 19:50:33.286158 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.286108    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b251c64b-ea33-428c-8664-1c80333cace9-openshift-service-ca\") pod \"maas-api-5cb9864d77-trrxz\" (UID: \"b251c64b-ea33-428c-8664-1c80333cace9\") " pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"
Sep 06 19:50:33.286206 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.286159    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b251c64b-ea33-428c-8664-1c80333cace9-metrics-certs\") pod \"maas-api-5cb9864d77-trrxz\" (UID: \"b251c64b-ea33-428c-8664-1c80333cace9\") " pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"
Sep 06 19:50:33.288514 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.288491    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 06 19:50:33.288721 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.288700    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 06 19:50:33.288852 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.288798    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 19:50:33.288954 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.288534    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gfj4k\""
Sep 06 19:50:33.289065 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.288631    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 06 19:50:33.296520 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.296499    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"]
Sep 06 19:50:33.300950 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.300923    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d767997f7-zbrr4"]
Sep 06 19:50:33.305121 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.305096    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d767997f7-zbrr4"
Sep 06 19:50:33.310246 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.310224    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d767997f7-zbrr4"]
Sep 06 19:50:33.386744 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.386714    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkbkz\" (UniqueName: \"kubernetes.io/projected/414e0638-cf42-43bc-92cd-0d5467b67ac2-kube-api-access-lkbkz\") pod \"maas-api-6d767997f7-zbrr4\" (UID: \"414e0638-cf42-43bc-92cd-0d5467b67ac2\") " pod="odh-ai-gateway-infra/maas-api-6d767997f7-zbrr4"
Sep 06 19:50:33.386901 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.386759    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b251c64b-ea33-428c-8664-1c80333cace9-maas-api-tls\") pod \"maas-api-5cb9864d77-trrxz\" (UID: \"b251c64b-ea33-428c-8664-1c80333cace9\") " pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"
Sep 06 19:50:33.386901 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.386800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b251c64b-ea33-428c-8664-1c80333cace9-openshift-service-ca\") pod \"maas-api-5cb9864d77-trrxz\" (UID: \"b251c64b-ea33-428c-8664-1c80333cace9\") " pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"
Sep 06 19:50:33.386901 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.386825    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414e0638-cf42-43bc-92cd-0d5467b67ac2-metrics-certs\") pod \"maas-api-6d767997f7-zbrr4\" (UID: \"414e0638-cf42-43bc-92cd-0d5467b67ac2\") " pod="odh-ai-gateway-infra/maas-api-6d767997f7-zbrr4"
Sep 06 19:50:33.386901 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.386862    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b251c64b-ea33-428c-8664-1c80333cace9-metrics-certs\") pod \"maas-api-5cb9864d77-trrxz\" (UID: \"b251c64b-ea33-428c-8664-1c80333cace9\") " pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"
Sep 06 19:50:33.387062 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.386914    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414e0638-cf42-43bc-92cd-0d5467b67ac2-openshift-service-ca\") pod \"maas-api-6d767997f7-zbrr4\" (UID: \"414e0638-cf42-43bc-92cd-0d5467b67ac2\") " pod="odh-ai-gateway-infra/maas-api-6d767997f7-zbrr4"
Sep 06 19:50:33.387062 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.386967    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414e0638-cf42-43bc-92cd-0d5467b67ac2-maas-api-tls\") pod \"maas-api-6d767997f7-zbrr4\" (UID: \"414e0638-cf42-43bc-92cd-0d5467b67ac2\") " pod="odh-ai-gateway-infra/maas-api-6d767997f7-zbrr4"
Sep 06 19:50:33.387062 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.387015    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-992dv\" (UniqueName: \"kubernetes.io/projected/b251c64b-ea33-428c-8664-1c80333cace9-kube-api-access-992dv\") pod \"maas-api-5cb9864d77-trrxz\" (UID: \"b251c64b-ea33-428c-8664-1c80333cace9\") " pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"
Sep 06 19:50:33.387626 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.387512    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b251c64b-ea33-428c-8664-1c80333cace9-openshift-service-ca\") pod \"maas-api-5cb9864d77-trrxz\" (UID: \"b251c64b-ea33-428c-8664-1c80333cace9\") " pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"
Sep 06 19:50:33.389391 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.389372    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b251c64b-ea33-428c-8664-1c80333cace9-maas-api-tls\") pod \"maas-api-5cb9864d77-trrxz\" (UID: \"b251c64b-ea33-428c-8664-1c80333cace9\") " pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"
Sep 06 19:50:33.389521 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.389502    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b251c64b-ea33-428c-8664-1c80333cace9-metrics-certs\") pod \"maas-api-5cb9864d77-trrxz\" (UID: \"b251c64b-ea33-428c-8664-1c80333cace9\") " pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"
Sep 06 19:50:33.394232 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.394212    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-992dv\" (UniqueName: \"kubernetes.io/projected/b251c64b-ea33-428c-8664-1c80333cace9-kube-api-access-992dv\") pod \"maas-api-5cb9864d77-trrxz\" (UID: \"b251c64b-ea33-428c-8664-1c80333cace9\") " pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"
Sep 06 19:50:33.487929 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.487903    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkbkz\" (UniqueName: \"kubernetes.io/projected/414e0638-cf42-43bc-92cd-0d5467b67ac2-kube-api-access-lkbkz\") pod \"maas-api-6d767997f7-zbrr4\" (UID: \"414e0638-cf42-43bc-92cd-0d5467b67ac2\") " pod="odh-ai-gateway-infra/maas-api-6d767997f7-zbrr4"
Sep 06 19:50:33.488037 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.487957    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414e0638-cf42-43bc-92cd-0d5467b67ac2-metrics-certs\") pod \"maas-api-6d767997f7-zbrr4\" (UID: \"414e0638-cf42-43bc-92cd-0d5467b67ac2\") " pod="odh-ai-gateway-infra/maas-api-6d767997f7-zbrr4"
Sep 06 19:50:33.488037 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.487999    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414e0638-cf42-43bc-92cd-0d5467b67ac2-openshift-service-ca\") pod \"maas-api-6d767997f7-zbrr4\" (UID: \"414e0638-cf42-43bc-92cd-0d5467b67ac2\") " pod="odh-ai-gateway-infra/maas-api-6d767997f7-zbrr4"
Sep 06 19:50:33.488131 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.488043    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414e0638-cf42-43bc-92cd-0d5467b67ac2-maas-api-tls\") pod \"maas-api-6d767997f7-zbrr4\" (UID: \"414e0638-cf42-43bc-92cd-0d5467b67ac2\") " pod="odh-ai-gateway-infra/maas-api-6d767997f7-zbrr4"
Sep 06 19:50:33.488691 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.488655    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414e0638-cf42-43bc-92cd-0d5467b67ac2-openshift-service-ca\") pod \"maas-api-6d767997f7-zbrr4\" (UID: \"414e0638-cf42-43bc-92cd-0d5467b67ac2\") " pod="odh-ai-gateway-infra/maas-api-6d767997f7-zbrr4"
Sep 06 19:50:33.490201 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.490176    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414e0638-cf42-43bc-92cd-0d5467b67ac2-metrics-certs\") pod \"maas-api-6d767997f7-zbrr4\" (UID: \"414e0638-cf42-43bc-92cd-0d5467b67ac2\") " pod="odh-ai-gateway-infra/maas-api-6d767997f7-zbrr4"
Sep 06 19:50:33.490359 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.490345    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414e0638-cf42-43bc-92cd-0d5467b67ac2-maas-api-tls\") pod \"maas-api-6d767997f7-zbrr4\" (UID: \"414e0638-cf42-43bc-92cd-0d5467b67ac2\") " pod="odh-ai-gateway-infra/maas-api-6d767997f7-zbrr4"
Sep 06 19:50:33.495289 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.495265    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkbkz\" (UniqueName: \"kubernetes.io/projected/414e0638-cf42-43bc-92cd-0d5467b67ac2-kube-api-access-lkbkz\") pod \"maas-api-6d767997f7-zbrr4\" (UID: \"414e0638-cf42-43bc-92cd-0d5467b67ac2\") " pod="odh-ai-gateway-infra/maas-api-6d767997f7-zbrr4"
Sep 06 19:50:33.600864 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.600786    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"
Sep 06 19:50:33.615356 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.615333    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d767997f7-zbrr4"
Sep 06 19:50:33.876069 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.876036    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"]
Sep 06 19:50:33.877351 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:50:33.877319    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb251c64b_ea33_428c_8664_1c80333cace9.slice/crio-41d34fd629807cae03283c644ab323077781b0d4be962714fd810b7f69bc9785 WatchSource:0}: Error finding container 41d34fd629807cae03283c644ab323077781b0d4be962714fd810b7f69bc9785: Status 404 returned error can't find the container with id 41d34fd629807cae03283c644ab323077781b0d4be962714fd810b7f69bc9785
Sep 06 19:50:33.878734 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.878715    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 19:50:33.896957 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:33.896934    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d767997f7-zbrr4"]
Sep 06 19:50:33.898718 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:50:33.898692    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414e0638_cf42_43bc_92cd_0d5467b67ac2.slice/crio-8d4a15ec2e2196b6157c5bec319bb76104ab8f4d0a47f27f72595da27bf8ef65 WatchSource:0}: Error finding container 8d4a15ec2e2196b6157c5bec319bb76104ab8f4d0a47f27f72595da27bf8ef65: Status 404 returned error can't find the container with id 8d4a15ec2e2196b6157c5bec319bb76104ab8f4d0a47f27f72595da27bf8ef65
Sep 06 19:50:34.598631 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:34.598502    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d767997f7-zbrr4" event={"ID":"414e0638-cf42-43bc-92cd-0d5467b67ac2","Type":"ContainerStarted","Data":"8d38b06600488ded8ed3aa00313e29f92fcf63ae1d1210d44c0d84ed76fc1309"}
Sep 06 19:50:34.599034 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:34.598708    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d767997f7-zbrr4"
Sep 06 19:50:34.599034 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:34.598735    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d767997f7-zbrr4" event={"ID":"414e0638-cf42-43bc-92cd-0d5467b67ac2","Type":"ContainerStarted","Data":"8d4a15ec2e2196b6157c5bec319bb76104ab8f4d0a47f27f72595da27bf8ef65"}
Sep 06 19:50:34.600106 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:34.600079    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz" event={"ID":"b251c64b-ea33-428c-8664-1c80333cace9","Type":"ContainerStarted","Data":"72c8f74ef0e2209e88513f23049f61479a7537bd8d5bdb1abcd39729f101803e"}
Sep 06 19:50:34.600213 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:34.600127    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"
Sep 06 19:50:34.600213 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:34.600143    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz" event={"ID":"b251c64b-ea33-428c-8664-1c80333cace9","Type":"ContainerStarted","Data":"41d34fd629807cae03283c644ab323077781b0d4be962714fd810b7f69bc9785"}
Sep 06 19:50:34.615436 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:34.615387    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d767997f7-zbrr4" podStartSLOduration=1.195327658 podStartE2EDuration="1.615372049s" totalImagesPullingTime="420.044391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:50:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:50:34.613497022 +0000 UTC m=+2368.794678830" watchObservedRunningTime="2026-09-06 19:50:34.615372049 +0000 UTC m=+2368.796553855"
Sep 06 19:50:34.630482 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:34.630433    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz" podStartSLOduration=1.16469607 podStartE2EDuration="1.630419559s" totalImagesPullingTime="465.723489ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:50:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:50:34.628326718 +0000 UTC m=+2368.809508540" watchObservedRunningTime="2026-09-06 19:50:34.630419559 +0000 UTC m=+2368.811601365"
Sep 06 19:50:40.611957 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:40.611928    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"
Sep 06 19:50:40.612363 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:40.612280    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d767997f7-zbrr4"
Sep 06 19:50:40.645811 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:40.645121    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:50:40.671879 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:40.671850    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"]
Sep 06 19:50:40.672039 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:40.672015    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz" podUID="b251c64b-ea33-428c-8664-1c80333cace9" containerName="maas-api" containerID="cri-o://72c8f74ef0e2209e88513f23049f61479a7537bd8d5bdb1abcd39729f101803e" gracePeriod=30
Sep 06 19:50:40.912862 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:40.912838    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"
Sep 06 19:50:40.948686 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:40.948656    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b251c64b-ea33-428c-8664-1c80333cace9-openshift-service-ca\") pod \"b251c64b-ea33-428c-8664-1c80333cace9\" (UID: \"b251c64b-ea33-428c-8664-1c80333cace9\") "
Sep 06 19:50:40.948797 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:40.948720    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b251c64b-ea33-428c-8664-1c80333cace9-maas-api-tls\") pod \"b251c64b-ea33-428c-8664-1c80333cace9\" (UID: \"b251c64b-ea33-428c-8664-1c80333cace9\") "
Sep 06 19:50:40.948857 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:40.948843    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-992dv\" (UniqueName: \"kubernetes.io/projected/b251c64b-ea33-428c-8664-1c80333cace9-kube-api-access-992dv\") pod \"b251c64b-ea33-428c-8664-1c80333cace9\" (UID: \"b251c64b-ea33-428c-8664-1c80333cace9\") "
Sep 06 19:50:40.948906 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:40.948894    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b251c64b-ea33-428c-8664-1c80333cace9-metrics-certs\") pod \"b251c64b-ea33-428c-8664-1c80333cace9\" (UID: \"b251c64b-ea33-428c-8664-1c80333cace9\") "
Sep 06 19:50:40.949009 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:40.948982    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b251c64b-ea33-428c-8664-1c80333cace9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b251c64b-ea33-428c-8664-1c80333cace9" (UID: "b251c64b-ea33-428c-8664-1c80333cace9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:50:40.949199 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:40.949177    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b251c64b-ea33-428c-8664-1c80333cace9-openshift-service-ca\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:50:40.950963 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:40.950934    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b251c64b-ea33-428c-8664-1c80333cace9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b251c64b-ea33-428c-8664-1c80333cace9" (UID: "b251c64b-ea33-428c-8664-1c80333cace9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:50:40.951060 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:40.950962    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b251c64b-ea33-428c-8664-1c80333cace9-kube-api-access-992dv" (OuterVolumeSpecName: "kube-api-access-992dv") pod "b251c64b-ea33-428c-8664-1c80333cace9" (UID: "b251c64b-ea33-428c-8664-1c80333cace9"). InnerVolumeSpecName "kube-api-access-992dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:50:40.951122 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:40.951060    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b251c64b-ea33-428c-8664-1c80333cace9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b251c64b-ea33-428c-8664-1c80333cace9" (UID: "b251c64b-ea33-428c-8664-1c80333cace9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:50:41.049673 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:41.049628    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-992dv\" (UniqueName: \"kubernetes.io/projected/b251c64b-ea33-428c-8664-1c80333cace9-kube-api-access-992dv\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:50:41.049673 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:41.049672    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b251c64b-ea33-428c-8664-1c80333cace9-metrics-certs\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:50:41.049849 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:41.049686    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b251c64b-ea33-428c-8664-1c80333cace9-maas-api-tls\") on node \"ip-10-0-129-252.ec2.internal\" DevicePath \"\""
Sep 06 19:50:41.641167 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:41.641131    2566 generic.go:356] "Generic (PLEG): container finished" podID="b251c64b-ea33-428c-8664-1c80333cace9" containerID="72c8f74ef0e2209e88513f23049f61479a7537bd8d5bdb1abcd39729f101803e" exitCode=0
Sep 06 19:50:41.641546 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:41.641177    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"
Sep 06 19:50:41.641546 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:41.641220    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz" event={"ID":"b251c64b-ea33-428c-8664-1c80333cace9","Type":"ContainerDied","Data":"72c8f74ef0e2209e88513f23049f61479a7537bd8d5bdb1abcd39729f101803e"}
Sep 06 19:50:41.641546 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:41.641271    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz" event={"ID":"b251c64b-ea33-428c-8664-1c80333cace9","Type":"ContainerDied","Data":"41d34fd629807cae03283c644ab323077781b0d4be962714fd810b7f69bc9785"}
Sep 06 19:50:41.641546 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:41.641293    2566 scope.go:117] "RemoveContainer" containerID="72c8f74ef0e2209e88513f23049f61479a7537bd8d5bdb1abcd39729f101803e"
Sep 06 19:50:41.651195 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:41.651178    2566 scope.go:117] "RemoveContainer" containerID="72c8f74ef0e2209e88513f23049f61479a7537bd8d5bdb1abcd39729f101803e"
Sep 06 19:50:41.651472 ip-10-0-129-252 kubenswrapper[2566]: E0906 19:50:41.651453    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72c8f74ef0e2209e88513f23049f61479a7537bd8d5bdb1abcd39729f101803e\": container with ID starting with 72c8f74ef0e2209e88513f23049f61479a7537bd8d5bdb1abcd39729f101803e not found: ID does not exist" containerID="72c8f74ef0e2209e88513f23049f61479a7537bd8d5bdb1abcd39729f101803e"
Sep 06 19:50:41.651531 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:41.651485    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72c8f74ef0e2209e88513f23049f61479a7537bd8d5bdb1abcd39729f101803e"} err="failed to get container status \"72c8f74ef0e2209e88513f23049f61479a7537bd8d5bdb1abcd39729f101803e\": rpc error: code = NotFound desc = could not find container \"72c8f74ef0e2209e88513f23049f61479a7537bd8d5bdb1abcd39729f101803e\": container with ID starting with 72c8f74ef0e2209e88513f23049f61479a7537bd8d5bdb1abcd39729f101803e not found: ID does not exist"
Sep 06 19:50:41.661673 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:41.661623    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"]
Sep 06 19:50:41.663794 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:41.663773    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb9864d77-trrxz"]
Sep 06 19:50:42.311077 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:42.311043    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b251c64b-ea33-428c-8664-1c80333cace9" path="/var/lib/kubelet/pods/b251c64b-ea33-428c-8664-1c80333cace9/volumes"
Sep 06 19:50:47.150061 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:47.149985    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:50:56.143442 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:50:56.143408    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:51:02.646777 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:02.646744    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-l5dmn"]
Sep 06 19:51:06.008416 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:06.008384    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9998564fb-k9b7t_fdb6e7f4-28c5-4486-a8f6-22788f72ea97/authorino/0.log"
Sep 06 19:51:06.544864 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:06.544841    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnssq_9254e6f1-b73d-49cc-8bc1-513925ccde4f/ovn-acl-logging/0.log"
Sep 06 19:51:06.553070 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:06.553051    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnssq_9254e6f1-b73d-49cc-8bc1-513925ccde4f/ovn-acl-logging/0.log"
Sep 06 19:51:11.782723 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:11.782692    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d4b9b44b9-wkxwn_e4b00d56-9a82-4e78-a700-49d155aeb7ac/copy-manifests/0.log"
Sep 06 19:51:11.788963 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:11.788940    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d4b9b44b9-wkxwn_e4b00d56-9a82-4e78-a700-49d155aeb7ac/manager/0.log"
Sep 06 19:51:12.342020 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:12.341987    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6d767997f7-zbrr4_414e0638-cf42-43bc-92cd-0d5467b67ac2/maas-api/0.log"
Sep 06 19:51:13.401290 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:13.401259    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9998564fb-k9b7t_fdb6e7f4-28c5-4486-a8f6-22788f72ea97/authorino/0.log"
Sep 06 19:51:13.724327 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:13.724298    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-xkhvn_6cb66892-087e-4585-af7d-3a56bf05144f/kuadrant-console-plugin/0.log"
Sep 06 19:51:13.965966 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:13.965936    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-l5dmn_af6ca84e-0ad3-4242-9918-5f0a4444245e/limitador/0.log"
Sep 06 19:51:14.750502 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:14.750473    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8b7764445-x6bd7_78ee1fa5-948c-4a27-a4a3-6934d802ed1b/kube-auth-proxy/0.log"
Sep 06 19:51:15.522387 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:15.522356    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-bfwq7_39a3ab5d-0bef-434b-83c9-6e48a52bf8e7/main/0.log"
Sep 06 19:51:21.449252 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:21.449216    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cqltr/must-gather-77ptf"]
Sep 06 19:51:21.449653 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:21.449639    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b251c64b-ea33-428c-8664-1c80333cace9" containerName="maas-api"
Sep 06 19:51:21.449702 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:21.449654    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b251c64b-ea33-428c-8664-1c80333cace9" containerName="maas-api"
Sep 06 19:51:21.449745 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:21.449736    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b251c64b-ea33-428c-8664-1c80333cace9" containerName="maas-api"
Sep 06 19:51:21.452902 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:21.452884    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cqltr/must-gather-77ptf"
Sep 06 19:51:21.455553 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:21.455513    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cqltr\"/\"openshift-service-ca.crt\""
Sep 06 19:51:21.456274 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:21.456250    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cqltr\"/\"default-dockercfg-dr2sd\""
Sep 06 19:51:21.456427 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:21.456283    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cqltr\"/\"kube-root-ca.crt\""
Sep 06 19:51:21.461501 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:21.461481    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cqltr/must-gather-77ptf"]
Sep 06 19:51:21.600965 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:21.600931    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/69311aca-3860-499c-9d8a-e49c3cb8a426-must-gather-output\") pod \"must-gather-77ptf\" (UID: \"69311aca-3860-499c-9d8a-e49c3cb8a426\") " pod="openshift-must-gather-cqltr/must-gather-77ptf"
Sep 06 19:51:21.601118 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:21.600973    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjz5n\" (UniqueName: \"kubernetes.io/projected/69311aca-3860-499c-9d8a-e49c3cb8a426-kube-api-access-xjz5n\") pod \"must-gather-77ptf\" (UID: \"69311aca-3860-499c-9d8a-e49c3cb8a426\") " pod="openshift-must-gather-cqltr/must-gather-77ptf"
Sep 06 19:51:21.701715 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:21.701690    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/69311aca-3860-499c-9d8a-e49c3cb8a426-must-gather-output\") pod \"must-gather-77ptf\" (UID: \"69311aca-3860-499c-9d8a-e49c3cb8a426\") " pod="openshift-must-gather-cqltr/must-gather-77ptf"
Sep 06 19:51:21.701865 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:21.701732    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjz5n\" (UniqueName: \"kubernetes.io/projected/69311aca-3860-499c-9d8a-e49c3cb8a426-kube-api-access-xjz5n\") pod \"must-gather-77ptf\" (UID: \"69311aca-3860-499c-9d8a-e49c3cb8a426\") " pod="openshift-must-gather-cqltr/must-gather-77ptf"
Sep 06 19:51:21.702034 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:21.702014    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/69311aca-3860-499c-9d8a-e49c3cb8a426-must-gather-output\") pod \"must-gather-77ptf\" (UID: \"69311aca-3860-499c-9d8a-e49c3cb8a426\") " pod="openshift-must-gather-cqltr/must-gather-77ptf"
Sep 06 19:51:21.711841 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:21.711818    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjz5n\" (UniqueName: \"kubernetes.io/projected/69311aca-3860-499c-9d8a-e49c3cb8a426-kube-api-access-xjz5n\") pod \"must-gather-77ptf\" (UID: \"69311aca-3860-499c-9d8a-e49c3cb8a426\") " pod="openshift-must-gather-cqltr/must-gather-77ptf"
Sep 06 19:51:21.763860 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:21.763827    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cqltr/must-gather-77ptf"
Sep 06 19:51:22.036761 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:22.036739    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cqltr/must-gather-77ptf"]
Sep 06 19:51:22.038737 ip-10-0-129-252 kubenswrapper[2566]: W0906 19:51:22.038712    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69311aca_3860_499c_9d8a_e49c3cb8a426.slice/crio-d02999a58fbea9afad113fd45117e9bbf5f9373f993a82812e25648d085fa3e7 WatchSource:0}: Error finding container d02999a58fbea9afad113fd45117e9bbf5f9373f993a82812e25648d085fa3e7: Status 404 returned error can't find the container with id d02999a58fbea9afad113fd45117e9bbf5f9373f993a82812e25648d085fa3e7
Sep 06 19:51:22.864736 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:22.864522    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cqltr/must-gather-77ptf" event={"ID":"69311aca-3860-499c-9d8a-e49c3cb8a426","Type":"ContainerStarted","Data":"73e1ff5e7e9cfba00c5e77a1eff5d043b95c68537665e3e2bb003eed3e5995bc"}
Sep 06 19:51:22.864736 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:22.864580    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cqltr/must-gather-77ptf" event={"ID":"69311aca-3860-499c-9d8a-e49c3cb8a426","Type":"ContainerStarted","Data":"d02999a58fbea9afad113fd45117e9bbf5f9373f993a82812e25648d085fa3e7"}
Sep 06 19:51:23.873422 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:23.873376    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cqltr/must-gather-77ptf" event={"ID":"69311aca-3860-499c-9d8a-e49c3cb8a426","Type":"ContainerStarted","Data":"7653cc3af4860e9a449e131059b5e603d4867d7d7247c344092b0c4da1421894"}
Sep 06 19:51:23.889094 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:23.889041    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cqltr/must-gather-77ptf" podStartSLOduration=2.192557567 podStartE2EDuration="2.889025112s" totalImagesPullingTime="696.467545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:51:23.887291973 +0000 UTC m=+2418.068473804" watchObservedRunningTime="2026-09-06 19:51:23.889025112 +0000 UTC m=+2418.070206962"
Sep 06 19:51:24.212927 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:24.212883    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7crp9_dcd6e3bb-3d92-4fe9-8432-62ef628b2472/global-pull-secret-syncer/0.log"
Sep 06 19:51:24.357273 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:24.357233    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b5vzh_30fe2339-bf82-4be5-8cf4-f68f98014b4f/konnectivity-agent/0.log"
Sep 06 19:51:24.404596 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:24.404567    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-252.ec2.internal_faae1a37c3a006a83d2995caac1d3848/haproxy/0.log"
Sep 06 19:51:29.249236 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:29.248045    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9998564fb-k9b7t_fdb6e7f4-28c5-4486-a8f6-22788f72ea97/authorino/0.log"
Sep 06 19:51:29.327989 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:29.327956    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-xkhvn_6cb66892-087e-4585-af7d-3a56bf05144f/kuadrant-console-plugin/0.log"
Sep 06 19:51:29.396077 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:29.396024    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-l5dmn_af6ca84e-0ad3-4242-9918-5f0a4444245e/limitador/0.log"
Sep 06 19:51:30.884335 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:30.884301    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a730856-a0dc-4cfe-895f-a973ac07f2db/alertmanager/0.log"
Sep 06 19:51:30.904055 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:30.904022    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a730856-a0dc-4cfe-895f-a973ac07f2db/config-reloader/0.log"
Sep 06 19:51:30.921873 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:30.921841    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a730856-a0dc-4cfe-895f-a973ac07f2db/kube-rbac-proxy-web/0.log"
Sep 06 19:51:30.943174 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:30.943152    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a730856-a0dc-4cfe-895f-a973ac07f2db/kube-rbac-proxy/0.log"
Sep 06 19:51:30.962511 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:30.962485    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a730856-a0dc-4cfe-895f-a973ac07f2db/kube-rbac-proxy-metric/0.log"
Sep 06 19:51:30.985830 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:30.985807    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a730856-a0dc-4cfe-895f-a973ac07f2db/prom-label-proxy/0.log"
Sep 06 19:51:31.000657 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:31.000629    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a730856-a0dc-4cfe-895f-a973ac07f2db/init-config-reloader/0.log"
Sep 06 19:51:31.040345 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:31.040311    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-59sch_ffa3473d-8821-4d6b-b633-c9b7a28f7285/cluster-monitoring-operator/0.log"
Sep 06 19:51:31.191007 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:31.190917    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c8h6d_140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c/node-exporter/0.log"
Sep 06 19:51:31.212543 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:31.212518    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c8h6d_140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c/kube-rbac-proxy/0.log"
Sep 06 19:51:31.236887 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:31.236843    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c8h6d_140b4a5c-ce0b-40ae-b4eb-26f17ef8d00c/init-textfile/0.log"
Sep 06 19:51:31.419795 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:31.419763    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-5vf4h_27cdabc3-8f8e-400d-a868-7fe028fdffe7/kube-rbac-proxy-main/0.log"
Sep 06 19:51:31.441526 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:31.441450    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-5vf4h_27cdabc3-8f8e-400d-a868-7fe028fdffe7/kube-rbac-proxy-self/0.log"
Sep 06 19:51:31.465593 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:31.462962    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-5vf4h_27cdabc3-8f8e-400d-a868-7fe028fdffe7/openshift-state-metrics/0.log"
Sep 06 19:51:31.682812 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:31.682774    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-jwt4v_fb73d373-f92d-4c73-bb27-61e4192eb39b/prometheus-operator/0.log"
Sep 06 19:51:31.697789 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:31.697706    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-jwt4v_fb73d373-f92d-4c73-bb27-61e4192eb39b/kube-rbac-proxy/0.log"
Sep 06 19:51:31.750708 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:31.750633    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-547fddd874-2z2wm_a5a629c9-0744-427c-8e88-1d4e2be4c7a9/telemeter-client/0.log"
Sep 06 19:51:31.771746 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:31.771698    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-547fddd874-2z2wm_a5a629c9-0744-427c-8e88-1d4e2be4c7a9/reload/0.log"
Sep 06 19:51:31.796765 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:31.796730    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-547fddd874-2z2wm_a5a629c9-0744-427c-8e88-1d4e2be4c7a9/kube-rbac-proxy/0.log"
Sep 06 19:51:34.406019 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:34.405975    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2"]
Sep 06 19:51:34.414685 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:34.414658    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2"
Sep 06 19:51:34.419947 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:34.419923    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2"]
Sep 06 19:51:34.533840 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:34.533795    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-d67f7658d-94bzn_1bb3e397-280b-49f1-bb38-c6c351489c10/console/0.log"
Sep 06 19:51:34.537539 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:34.537512    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a23ebdcd-87f2-477b-a6ac-8241e03d00d4-lib-modules\") pod \"perf-node-gather-daemonset-ls9k2\" (UID: \"a23ebdcd-87f2-477b-a6ac-8241e03d00d4\") " pod="openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2"
Sep 06 19:51:34.537697 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:34.537562    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gkr6\" (UniqueName: \"kubernetes.io/projected/a23ebdcd-87f2-477b-a6ac-8241e03d00d4-kube-api-access-8gkr6\") pod \"perf-node-gather-daemonset-ls9k2\" (UID: \"a23ebdcd-87f2-477b-a6ac-8241e03d00d4\") " pod="openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2"
Sep 06 19:51:34.537697 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:34.537637    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a23ebdcd-87f2-477b-a6ac-8241e03d00d4-proc\") pod \"perf-node-gather-daemonset-ls9k2\" (UID: \"a23ebdcd-87f2-477b-a6ac-8241e03d00d4\") " pod="openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2"
Sep 06 19:51:34.537697 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:34.537676    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a23ebdcd-87f2-477b-a6ac-8241e03d00d4-sys\") pod \"perf-node-gather-daemonset-ls9k2\" (UID: \"a23ebdcd-87f2-477b-a6ac-8241e03d00d4\") " pod="openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2"
Sep 06 19:51:34.537910 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:34.537699    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a23ebdcd-87f2-477b-a6ac-8241e03d00d4-podres\") pod \"perf-node-gather-daemonset-ls9k2\" (UID: \"a23ebdcd-87f2-477b-a6ac-8241e03d00d4\") " pod="openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2"
Sep 06 19:51:34.638833 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:34.638786    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a23ebdcd-87f2-477b-a6ac-8241e03d00d4-sys\") pod \"perf-node-gather-daemonset-ls9k2\" (UID: \"a23ebdcd-87f2-477b-a6ac-8241e03d00d4\") " pod="openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2"
Sep 06 19:51:34.639024 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:34.638845    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a23ebdcd-87f2-477b-a6ac-8241e03d00d4-podres\") pod \"perf-node-gather-daemonset-ls9k2\" (UID: \"a23ebdcd-87f2-477b-a6ac-8241e03d00d4\") " pod="openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2"
Sep 06 19:51:34.639024 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:34.638891    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a23ebdcd-87f2-477b-a6ac-8241e03d00d4-lib-modules\") pod \"perf-node-gather-daemonset-ls9k2\" (UID: \"a23ebdcd-87f2-477b-a6ac-8241e03d00d4\") " pod="openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2"
Sep 06 19:51:34.639024 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:34.638940    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gkr6\" (UniqueName: \"kubernetes.io/projected/a23ebdcd-87f2-477b-a6ac-8241e03d00d4-kube-api-access-8gkr6\") pod \"perf-node-gather-daemonset-ls9k2\" (UID: \"a23ebdcd-87f2-477b-a6ac-8241e03d00d4\") " pod="openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2"
Sep 06 19:51:34.639024 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:34.638983    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a23ebdcd-87f2-477b-a6ac-8241e03d00d4-proc\") pod \"perf-node-gather-daemonset-ls9k2\" (UID: \"a23ebdcd-87f2-477b-a6ac-8241e03d00d4\") " pod="openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2"
Sep 06 19:51:34.639248 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:34.639105    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a23ebdcd-87f2-477b-a6ac-8241e03d00d4-proc\") pod \"perf-node-gather-daemonset-ls9k2\" (UID: \"a23ebdcd-87f2-477b-a6ac-8241e03d00d4\") " pod="openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2"
Sep 06 19:51:34.639248 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:34.639164    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a23ebdcd-87f2-477b-a6ac-8241e03d00d4-sys\") pod \"perf-node-gather-daemonset-ls9k2\" (UID: \"a23ebdcd-87f2-477b-a6ac-8241e03d00d4\") " pod="openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2"
Sep 06 19:51:34.639347 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:34.639272    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a23ebdcd-87f2-477b-a6ac-8241e03d00d4-podres\") pod \"perf-node-gather-daemonset-ls9k2\" (UID: \"a23ebdcd-87f2-477b-a6ac-8241e03d00d4\") " pod="openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2"
Sep 06 19:51:34.639406 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:34.639384    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a23ebdcd-87f2-477b-a6ac-8241e03d00d4-lib-modules\") pod \"perf-node-gather-daemonset-ls9k2\" (UID: \"a23ebdcd-87f2-477b-a6ac-8241e03d00d4\") " pod="openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2"
Sep 06 19:51:34.647955 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:34.647923    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gkr6\" (UniqueName: \"kubernetes.io/projected/a23ebdcd-87f2-477b-a6ac-8241e03d00d4-kube-api-access-8gkr6\") pod \"perf-node-gather-daemonset-ls9k2\" (UID: \"a23ebdcd-87f2-477b-a6ac-8241e03d00d4\") " pod="openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2"
Sep 06 19:51:34.729135 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:34.729099    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2"
Sep 06 19:51:35.017456 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:35.017407    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2"]
Sep 06 19:51:35.225724 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:35.225695    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-lh7gj_54d49606-bc52-481e-a3f7-bf0f3b60ab2c/volume-data-source-validator/0.log"
Sep 06 19:51:35.973783 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:35.973739    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2" event={"ID":"a23ebdcd-87f2-477b-a6ac-8241e03d00d4","Type":"ContainerStarted","Data":"e844f0832a0d13f4dd00bf67a5ca56a452fec40a7eb2d271ed2e16594d482c76"}
Sep 06 19:51:35.974273 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:35.973812    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2" event={"ID":"a23ebdcd-87f2-477b-a6ac-8241e03d00d4","Type":"ContainerStarted","Data":"1cb70ce3e6fc440eeea4dd1440aaa4f21b623e7406d9825f54d5ab46ad501a5b"}
Sep 06 19:51:35.974273 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:35.974064    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2"
Sep 06 19:51:35.988760 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:35.988710    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2" podStartSLOduration=1.988692364 podStartE2EDuration="1.988692364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:51:35.987392703 +0000 UTC m=+2430.168574512" watchObservedRunningTime="2026-09-06 19:51:35.988692364 +0000 UTC m=+2430.169874171"
Sep 06 19:51:36.443481 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:36.443397    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gqxrr_a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88/dns/0.log"
Sep 06 19:51:36.461001 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:36.460974    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gqxrr_a2376ae5-11e6-4a7f-9e5d-9f75a0c43f88/kube-rbac-proxy/0.log"
Sep 06 19:51:36.498958 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:36.498932    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jm5gt_176abf0b-9b68-46d3-a657-ab219939488a/dns-node-resolver/0.log"
Sep 06 19:51:37.039709 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:37.039678    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6dcf6d58c-nd6k8_22089faf-55a4-4003-849a-56580918f07b/registry/0.log"
Sep 06 19:51:37.101471 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:37.101437    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x75tt_a68be042-a345-44a5-9564-1d6d16d11b8a/node-ca/0.log"
Sep 06 19:51:38.299634 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:38.299584    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8b7764445-x6bd7_78ee1fa5-948c-4a27-a4a3-6934d802ed1b/kube-auth-proxy/0.log"
Sep 06 19:51:39.259724 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:39.259694    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n9qkv_143de454-6f38-4836-a21c-5cac7ad78381/serve-healthcheck-canary/0.log"
Sep 06 19:51:39.703480 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:39.703438    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-djlz4_033d0561-8f3c-4e3c-afad-88da2108830e/insights-operator/0.log"
Sep 06 19:51:39.705945 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:39.705918    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-djlz4_033d0561-8f3c-4e3c-afad-88da2108830e/insights-operator/1.log"
Sep 06 19:51:39.844573 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:39.844545    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kjj7s_ede8a3c0-14ed-4a64-beda-caa3b4c5d601/kube-rbac-proxy/0.log"
Sep 06 19:51:39.862499 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:39.862475    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kjj7s_ede8a3c0-14ed-4a64-beda-caa3b4c5d601/exporter/0.log"
Sep 06 19:51:39.880732 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:39.880704    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kjj7s_ede8a3c0-14ed-4a64-beda-caa3b4c5d601/extractor/0.log"
Sep 06 19:51:41.992729 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:41.992700    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cqltr/perf-node-gather-daemonset-ls9k2"
Sep 06 19:51:42.456102 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:42.456063    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d4b9b44b9-wkxwn_e4b00d56-9a82-4e78-a700-49d155aeb7ac/manager/0.log"
Sep 06 19:51:42.472517 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:42.472489    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d4b9b44b9-wkxwn_e4b00d56-9a82-4e78-a700-49d155aeb7ac/copy-manifests/0.log"
Sep 06 19:51:43.994271 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:43.994234    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5c4c9c58b7-hfv74_b98aafd2-cccd-41a8-abce-c8ee82e5a972/manager/0.log"
Sep 06 19:51:45.102354 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:45.102327    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8b7764445-x6bd7_78ee1fa5-948c-4a27-a4a3-6934d802ed1b/kube-auth-proxy/0.log"
Sep 06 19:51:46.982059 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:46.982028    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8b7764445-x6bd7_78ee1fa5-948c-4a27-a4a3-6934d802ed1b/kube-auth-proxy/0.log"
Sep 06 19:51:50.658380 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:50.658355    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4ncl_0b554b5b-8ce8-4674-b7d3-235a7c0b777b/kube-multus-additional-cni-plugins/0.log"
Sep 06 19:51:50.679714 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:50.679693    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4ncl_0b554b5b-8ce8-4674-b7d3-235a7c0b777b/egress-router-binary-copy/0.log"
Sep 06 19:51:50.697222 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:50.697194    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4ncl_0b554b5b-8ce8-4674-b7d3-235a7c0b777b/cni-plugins/0.log"
Sep 06 19:51:50.715515 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:50.715489    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4ncl_0b554b5b-8ce8-4674-b7d3-235a7c0b777b/bond-cni-plugin/0.log"
Sep 06 19:51:50.735537 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:50.735505    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4ncl_0b554b5b-8ce8-4674-b7d3-235a7c0b777b/routeoverride-cni/0.log"
Sep 06 19:51:50.752253 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:50.752231    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4ncl_0b554b5b-8ce8-4674-b7d3-235a7c0b777b/whereabouts-cni-bincopy/0.log"
Sep 06 19:51:50.770017 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:50.769995    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4ncl_0b554b5b-8ce8-4674-b7d3-235a7c0b777b/whereabouts-cni/0.log"
Sep 06 19:51:50.871754 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:50.871713    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-tb692_a21c0317-3ef0-4b2d-ae52-809097383928/kube-multus/0.log"
Sep 06 19:51:50.928011 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:50.927916    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qlvsw_83a7379b-ab58-4439-998e-448e48150f3b/network-metrics-daemon/0.log"
Sep 06 19:51:50.941106 ip-10-0-129-252 kubenswrapper[2566]: I0906 19:51:50.941083    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qlvsw_83a7379b-ab58-4439-998e-448e48150f3b/kube-rbac-proxy/0.log"