Sep 08 20:25:46.640552 ip-10-0-129-118 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 20:25:46.994458 ip-10-0-129-118 kubenswrapper[2552]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 20:25:46.994458 ip-10-0-129-118 kubenswrapper[2552]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 20:25:46.994458 ip-10-0-129-118 kubenswrapper[2552]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 20:25:46.994458 ip-10-0-129-118 kubenswrapper[2552]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 20:25:46.994458 ip-10-0-129-118 kubenswrapper[2552]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 20:25:46.995519 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:46.995429    2552 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 20:25:46.998334 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998318    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 20:25:46.998334 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998333    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 20:25:46.998392 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998337    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 20:25:46.998392 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998340    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 20:25:46.998392 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998343    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 20:25:46.998392 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998345    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 20:25:46.998392 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998348    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 20:25:46.998392 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998351    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 20:25:46.998392 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998354    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 20:25:46.998392 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998356    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 20:25:46.998392 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998359    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 20:25:46.998392 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998361    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 20:25:46.998392 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998364    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 20:25:46.998392 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998367    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 20:25:46.998392 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998369    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 20:25:46.998392 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998372    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 20:25:46.998392 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998375    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 20:25:46.998392 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998377    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 20:25:46.998392 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998380    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 20:25:46.998392 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998384    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 20:25:46.998392 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998386    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 20:25:46.998392 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998389    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 20:25:46.998392 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998392    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 20:25:46.998893 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998394    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 20:25:46.998893 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998398    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 20:25:46.998893 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998401    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 20:25:46.998893 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998404    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 20:25:46.998893 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998414    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 20:25:46.998893 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998417    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 20:25:46.998893 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998420    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 20:25:46.998893 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998422    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 20:25:46.998893 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998425    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 20:25:46.998893 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998427    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 20:25:46.998893 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998430    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 20:25:46.998893 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998432    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 20:25:46.998893 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998435    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 20:25:46.998893 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998437    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 20:25:46.998893 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998440    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 20:25:46.998893 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998442    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 20:25:46.998893 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998445    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 20:25:46.998893 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998447    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 20:25:46.998893 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998450    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 20:25:46.999412 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998452    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 20:25:46.999412 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998455    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 20:25:46.999412 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998457    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 20:25:46.999412 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998460    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 20:25:46.999412 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998462    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 20:25:46.999412 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998465    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 20:25:46.999412 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998467    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 20:25:46.999412 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998470    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 20:25:46.999412 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998474    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 20:25:46.999412 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998478    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 20:25:46.999412 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998482    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 20:25:46.999412 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998485    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 20:25:46.999412 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998488    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 20:25:46.999412 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998490    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 20:25:46.999412 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998494    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 20:25:46.999412 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998497    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 20:25:46.999412 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998500    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 20:25:46.999412 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998503    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 20:25:46.999412 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998505    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 20:25:46.999894 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998508    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 20:25:46.999894 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998510    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 20:25:46.999894 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998513    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 20:25:46.999894 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998516    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 20:25:46.999894 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998519    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 20:25:46.999894 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998521    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 20:25:46.999894 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998524    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 20:25:46.999894 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998527    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 20:25:46.999894 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998530    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 20:25:46.999894 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998532    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 20:25:46.999894 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998535    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 20:25:46.999894 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998539    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 20:25:46.999894 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998542    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 20:25:46.999894 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998545    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 20:25:46.999894 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998548    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 20:25:46.999894 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998551    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 20:25:46.999894 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998553    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 20:25:46.999894 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998556    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 20:25:46.999894 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998559    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 20:25:47.000374 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998563    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 20:25:47.000374 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998567    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 20:25:47.000374 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998571    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 20:25:47.000374 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998573    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 20:25:47.000374 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998576    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 20:25:47.000374 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998579    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 20:25:47.000374 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998582    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 20:25:47.000374 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998585    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 20:25:47.000374 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998588    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 20:25:47.000374 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998591    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 20:25:47.000374 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998595    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 20:25:47.000374 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998598    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 20:25:47.000374 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998601    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 20:25:47.000374 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998603    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 20:25:47.000374 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998606    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 20:25:47.000374 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998609    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 20:25:47.000374 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998611    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 20:25:47.000374 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998614    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 20:25:47.000374 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:46.998616    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 20:25:47.000374 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000034    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 20:25:47.000874 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000044    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 20:25:47.000874 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000047    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 20:25:47.000874 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000050    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 20:25:47.000874 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000053    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 20:25:47.000874 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000056    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 20:25:47.000874 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000059    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 20:25:47.000874 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000062    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 20:25:47.000874 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000064    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 20:25:47.000874 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000067    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 20:25:47.000874 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000069    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 20:25:47.000874 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000072    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 20:25:47.000874 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000074    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 20:25:47.000874 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000077    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 20:25:47.000874 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000080    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 20:25:47.000874 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000083    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 20:25:47.000874 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000086    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 20:25:47.000874 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000088    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 20:25:47.000874 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000091    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 20:25:47.000874 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000093    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 20:25:47.001353 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000096    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 20:25:47.001353 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000100    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 20:25:47.001353 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000103    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 20:25:47.001353 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000105    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 20:25:47.001353 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000108    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 20:25:47.001353 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000111    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 20:25:47.001353 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000113    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 20:25:47.001353 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000116    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 20:25:47.001353 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000119    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 20:25:47.001353 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000121    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 20:25:47.001353 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000124    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 20:25:47.001353 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000127    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 20:25:47.001353 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000129    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 20:25:47.001353 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000132    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 20:25:47.001353 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000135    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 20:25:47.001353 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000138    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 20:25:47.001353 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000140    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 20:25:47.001353 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000144    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 20:25:47.001353 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000148    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 20:25:47.001353 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000151    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 20:25:47.001898 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000153    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 20:25:47.001898 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000156    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 20:25:47.001898 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000159    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 20:25:47.001898 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000162    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 20:25:47.001898 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000164    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 20:25:47.001898 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000167    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 20:25:47.001898 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000169    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 20:25:47.001898 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000173    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 20:25:47.001898 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000175    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 20:25:47.001898 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000178    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 20:25:47.001898 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000180    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 20:25:47.001898 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000183    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 20:25:47.001898 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000185    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 20:25:47.001898 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000189    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 20:25:47.001898 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000192    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 20:25:47.001898 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000194    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 20:25:47.001898 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000197    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 20:25:47.001898 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000199    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 20:25:47.001898 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000202    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 20:25:47.002386 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000205    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 20:25:47.002386 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000209    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 20:25:47.002386 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000211    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 20:25:47.002386 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000214    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 20:25:47.002386 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000217    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 20:25:47.002386 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000219    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 20:25:47.002386 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000222    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 20:25:47.002386 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000224    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 20:25:47.002386 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000227    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 20:25:47.002386 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000229    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 20:25:47.002386 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000232    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 20:25:47.002386 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000249    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 20:25:47.002386 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000252    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 20:25:47.002386 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000257    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 20:25:47.002386 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000260    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 20:25:47.002386 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000264    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 20:25:47.002386 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000268    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 20:25:47.002386 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000271    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 20:25:47.002386 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000274    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 20:25:47.002867 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000277    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 20:25:47.002867 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000279    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 20:25:47.002867 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000284    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 20:25:47.002867 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000288    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 20:25:47.002867 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000291    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 20:25:47.002867 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000294    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 20:25:47.002867 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000297    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 20:25:47.002867 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000300    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 20:25:47.002867 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000303    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 20:25:47.002867 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000306    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 20:25:47.002867 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000308    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 20:25:47.002867 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000311    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 20:25:47.002867 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000314    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 20:25:47.002867 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000317    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 20:25:47.002867 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000320    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 20:25:47.002867 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000322    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 20:25:47.002867 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000325    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 20:25:47.002867 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000328    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 20:25:47.002867 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000330    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 20:25:47.002867 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000332    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 20:25:47.003375 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000335    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 20:25:47.003375 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000734    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 20:25:47.003375 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000741    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 20:25:47.003375 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000745    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 20:25:47.003375 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000749    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 20:25:47.003375 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000752    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 20:25:47.003375 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000755    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 20:25:47.003375 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000758    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 20:25:47.003375 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000761    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 20:25:47.003375 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000763    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 20:25:47.003375 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000766    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 20:25:47.003375 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000769    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 20:25:47.003375 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000772    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 20:25:47.003375 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000775    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 20:25:47.003375 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000779    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 20:25:47.003375 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000783    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 20:25:47.003375 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000787    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 20:25:47.003375 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000790    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 20:25:47.003375 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000793    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 20:25:47.003845 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000796    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 20:25:47.003845 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000798    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 20:25:47.003845 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000801    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 20:25:47.003845 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000804    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 20:25:47.003845 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000806    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 20:25:47.003845 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000809    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 20:25:47.003845 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000812    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 20:25:47.003845 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000814    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 20:25:47.003845 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000817    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 20:25:47.003845 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000820    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 20:25:47.003845 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000823    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 20:25:47.003845 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000825    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 20:25:47.003845 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000828    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 20:25:47.003845 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000831    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 20:25:47.003845 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000833    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 20:25:47.003845 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000837    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 20:25:47.003845 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000840    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 20:25:47.003845 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000842    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 20:25:47.003845 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000844    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 20:25:47.003845 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000847    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 20:25:47.004390 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000850    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 20:25:47.004390 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000852    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 20:25:47.004390 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000857    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 20:25:47.004390 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000860    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 20:25:47.004390 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000863    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 20:25:47.004390 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000865    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 20:25:47.004390 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000868    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 20:25:47.004390 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000870    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 20:25:47.004390 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000873    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 20:25:47.004390 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000876    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 20:25:47.004390 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000878    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 20:25:47.004390 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000881    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 20:25:47.004390 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000884    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 20:25:47.004390 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000887    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 20:25:47.004390 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000889    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 20:25:47.004390 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000892    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 20:25:47.004390 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000895    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 20:25:47.004390 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000897    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 20:25:47.004390 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000900    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 20:25:47.004390 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000902    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 20:25:47.004889 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000905    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 20:25:47.004889 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000907    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 20:25:47.004889 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000910    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 20:25:47.004889 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000912    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 20:25:47.004889 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000915    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 20:25:47.004889 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000917    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 20:25:47.004889 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000920    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 20:25:47.004889 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000924    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 20:25:47.004889 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000927    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 20:25:47.004889 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000930    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 20:25:47.004889 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000932    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 20:25:47.004889 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000934    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 20:25:47.004889 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000937    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 20:25:47.004889 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000939    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 20:25:47.004889 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000945    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 20:25:47.004889 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000947    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 20:25:47.004889 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000950    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 20:25:47.004889 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000953    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 20:25:47.004889 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000955    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 20:25:47.005371 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000958    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 20:25:47.005371 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000960    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 20:25:47.005371 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000967    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 20:25:47.005371 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000970    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 20:25:47.005371 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000972    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 20:25:47.005371 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000975    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 20:25:47.005371 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000977    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 20:25:47.005371 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000980    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 20:25:47.005371 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000982    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 20:25:47.005371 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000985    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 20:25:47.005371 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000989    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 20:25:47.005371 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000992    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 20:25:47.005371 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000995    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 20:25:47.005371 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.000998    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 20:25:47.005371 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001000    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 20:25:47.005371 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001003    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 20:25:47.005371 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001006    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 20:25:47.005371 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001008    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 20:25:47.005371 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001011    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 20:25:47.005371 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001013    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001017    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001020    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001083    2552 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001123    2552 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001136    2552 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001141    2552 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001144    2552 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001149    2552 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001153    2552 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001158    2552 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001161    2552 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001164    2552 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001167    2552 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001171    2552 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001174    2552 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001178    2552 flags.go:64] FLAG: --cgroup-root=""
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001181    2552 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001184    2552 flags.go:64] FLAG: --client-ca-file=""
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001187    2552 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001190    2552 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001196    2552 flags.go:64] FLAG: --cluster-domain=""
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001199    2552 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001202    2552 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 20:25:47.005852 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001206    2552 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001209    2552 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001212    2552 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001215    2552 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001218    2552 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001222    2552 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001225    2552 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001228    2552 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001230    2552 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001247    2552 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001250    2552 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001255    2552 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001258    2552 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001261    2552 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001264    2552 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001267    2552 flags.go:64] FLAG: --enable-server="true"
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001271    2552 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001276    2552 flags.go:64] FLAG: --event-burst="100"
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001279    2552 flags.go:64] FLAG: --event-qps="50"
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001282    2552 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001285    2552 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001288    2552 flags.go:64] FLAG: --eviction-hard=""
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001292    2552 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001295    2552 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001298    2552 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 20:25:47.006456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001301    2552 flags.go:64] FLAG: --eviction-soft=""
Sep 08 20:25:47.007064 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001304    2552 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 20:25:47.007064 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001307    2552 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 20:25:47.007064 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001310    2552 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 20:25:47.007064 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001313    2552 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 20:25:47.007064 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001316    2552 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 20:25:47.007064 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001319    2552 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 20:25:47.007064 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001321    2552 flags.go:64] FLAG: --feature-gates=""
Sep 08 20:25:47.007064 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001325    2552 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 20:25:47.007064 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001328    2552 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 20:25:47.007064 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001331    2552 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 20:25:47.007064 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001334    2552 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 20:25:47.007064 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001337    2552 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 20:25:47.007064 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001340    2552 flags.go:64] FLAG: --help="false"
Sep 08 20:25:47.007064 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001343    2552 flags.go:64] FLAG: --hostname-override="ip-10-0-129-118.ec2.internal"
Sep 08 20:25:47.007064 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001346    2552 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 20:25:47.007064 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001350    2552 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 20:25:47.007064 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001353    2552 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 20:25:47.007064 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001357    2552 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 20:25:47.007064 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001361    2552 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 20:25:47.007064 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001364    2552 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 20:25:47.007064 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001367    2552 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 20:25:47.007064 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001370    2552 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 20:25:47.007064 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001373    2552 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001376    2552 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001380    2552 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001383    2552 flags.go:64] FLAG: --kube-reserved=""
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001386    2552 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001388    2552 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001392    2552 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001394    2552 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001397    2552 flags.go:64] FLAG: --lock-file=""
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001400    2552 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001404    2552 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001407    2552 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001417    2552 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001419    2552 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001422    2552 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001425    2552 flags.go:64] FLAG: --logging-format="text"
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001428    2552 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001431    2552 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001434    2552 flags.go:64] FLAG: --manifest-url=""
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001437    2552 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001441    2552 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001443    2552 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001447    2552 flags.go:64] FLAG: --max-pods="110"
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001450    2552 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001453    2552 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 20:25:47.007722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001457    2552 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001460    2552 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001463    2552 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001466    2552 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001469    2552 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001476    2552 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001479    2552 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001483    2552 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001486    2552 flags.go:64] FLAG: --pod-cidr=""
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001489    2552 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001495    2552 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001498    2552 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001501    2552 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001504    2552 flags.go:64] FLAG: --port="10250"
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001507    2552 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001510    2552 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f92437555f9886db"
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001513    2552 flags.go:64] FLAG: --qos-reserved=""
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001516    2552 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001520    2552 flags.go:64] FLAG: --register-node="true"
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001523    2552 flags.go:64] FLAG: --register-with-taints=""
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001526    2552 flags.go:64] FLAG: --registry-burst="10"
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001529    2552 flags.go:64] FLAG: --registry-qps="5"
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001532    2552 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001535    2552 flags.go:64] FLAG: --reserved-memory=""
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001538    2552 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 20:25:47.008378 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001541    2552 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001544    2552 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001547    2552 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001550    2552 flags.go:64] FLAG: --runonce="false"
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001553    2552 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001555    2552 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001559    2552 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001563    2552 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001566    2552 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001569    2552 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001572    2552 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001576    2552 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001578    2552 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001582    2552 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001585    2552 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001587    2552 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001591    2552 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001594    2552 flags.go:64] FLAG: --system-cgroups=""
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001597    2552 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001602    2552 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001605    2552 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001607    2552 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001611    2552 flags.go:64] FLAG: --tls-min-version=""
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001615    2552 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001617    2552 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 20:25:47.009042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001620    2552 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 20:25:47.009688 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001624    2552 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 20:25:47.009688 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001626    2552 flags.go:64] FLAG: --v="2"
Sep 08 20:25:47.009688 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001630    2552 flags.go:64] FLAG: --version="false"
Sep 08 20:25:47.009688 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001634    2552 flags.go:64] FLAG: --vmodule=""
Sep 08 20:25:47.009688 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001638    2552 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 20:25:47.009688 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.001641    2552 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 20:25:47.009688 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001762    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 20:25:47.009688 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001765    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 20:25:47.009688 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001768    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 20:25:47.009688 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001771    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 20:25:47.009688 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001774    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 20:25:47.009688 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001776    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 20:25:47.009688 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001779    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 20:25:47.009688 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001783    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 20:25:47.009688 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001786    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 20:25:47.009688 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001788    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 20:25:47.009688 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001791    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 20:25:47.009688 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001793    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 20:25:47.009688 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001796    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 20:25:47.009688 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001799    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 20:25:47.009688 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001803    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 20:25:47.009688 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001807    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 20:25:47.010219 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001809    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 20:25:47.010219 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001812    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 20:25:47.010219 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001816    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 20:25:47.010219 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001820    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 20:25:47.010219 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001823    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 20:25:47.010219 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001825    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 20:25:47.010219 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001828    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 20:25:47.010219 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001830    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 20:25:47.010219 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001833    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 20:25:47.010219 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001835    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 20:25:47.010219 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001838    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 20:25:47.010219 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001842    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 20:25:47.010219 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001845    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 20:25:47.010219 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001848    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 20:25:47.010219 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001851    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 20:25:47.010219 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001854    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 20:25:47.010219 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001857    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 20:25:47.010219 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001869    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 20:25:47.010219 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001872    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 20:25:47.010704 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001875    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 20:25:47.010704 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001879    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 20:25:47.010704 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001883    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 20:25:47.010704 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001887    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 20:25:47.010704 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001893    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 20:25:47.010704 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001896    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 20:25:47.010704 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001901    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 20:25:47.010704 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001904    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 20:25:47.010704 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001906    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 20:25:47.010704 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001909    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 20:25:47.010704 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001912    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 20:25:47.010704 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001915    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 20:25:47.010704 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001918    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 20:25:47.010704 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001921    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 20:25:47.010704 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001924    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 20:25:47.010704 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001926    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 20:25:47.010704 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001929    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 20:25:47.010704 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001932    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 20:25:47.010704 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001934    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 20:25:47.011218 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001937    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 20:25:47.011218 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001940    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 20:25:47.011218 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001942    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 20:25:47.011218 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001945    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 20:25:47.011218 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001948    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 20:25:47.011218 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001950    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 20:25:47.011218 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001953    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 20:25:47.011218 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001956    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 20:25:47.011218 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001959    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 20:25:47.011218 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001961    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 20:25:47.011218 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001964    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 20:25:47.011218 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001966    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 20:25:47.011218 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001969    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 20:25:47.011218 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001971    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 20:25:47.011218 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001974    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 20:25:47.011218 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001976    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 20:25:47.011218 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001979    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 20:25:47.011218 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001982    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 20:25:47.011218 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001985    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 20:25:47.011701 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001987    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 20:25:47.011701 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001990    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 20:25:47.011701 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001992    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 20:25:47.011701 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001995    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 20:25:47.011701 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.001998    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 20:25:47.011701 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.002000    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 20:25:47.011701 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.002004    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 20:25:47.011701 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.002007    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 20:25:47.011701 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.002010    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 20:25:47.011701 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.002012    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 20:25:47.011701 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.002015    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 20:25:47.011701 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.002018    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 20:25:47.011701 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.002020    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 20:25:47.011701 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.002023    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 20:25:47.011701 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.002025    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 20:25:47.011701 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.002028    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 20:25:47.011701 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.002030    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 20:25:47.011701 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.002033    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 20:25:47.011701 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.002036    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 20:25:47.011701 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.002039    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 20:25:47.012379 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.002041    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 20:25:47.012379 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.002044    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 20:25:47.012379 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.002047    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 20:25:47.012379 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.002049    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 20:25:47.012379 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.002052    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 20:25:47.012379 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.002054    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 20:25:47.012379 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.002656    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 20:25:47.012379 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.010101    2552 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 20:25:47.012379 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.010116    2552 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 20:25:47.012379 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.011100    2552 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 20:25:47.012379 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.011110    2552 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 20:25:47.012379 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011166    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 20:25:47.012379 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011172    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 20:25:47.012379 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011175    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 20:25:47.012748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011178    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 20:25:47.012748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011181    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 20:25:47.012748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011184    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 20:25:47.012748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011188    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 20:25:47.012748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011193    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 20:25:47.012748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011196    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 20:25:47.012748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011199    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 20:25:47.012748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011202    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 20:25:47.012748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011205    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 20:25:47.012748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011208    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 20:25:47.012748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011211    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 20:25:47.012748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011214    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 20:25:47.012748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011216    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 20:25:47.012748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011219    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 20:25:47.012748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011222    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 20:25:47.012748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011224    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 20:25:47.012748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011227    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 20:25:47.012748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011230    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 20:25:47.012748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011233    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 20:25:47.012748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011248    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 20:25:47.013256 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011251    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 20:25:47.013256 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011254    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 20:25:47.013256 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011256    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 20:25:47.013256 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011259    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 20:25:47.013256 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011262    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 20:25:47.013256 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011264    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 20:25:47.013256 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011268    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 20:25:47.013256 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011271    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 20:25:47.013256 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011273    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 20:25:47.013256 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011276    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 20:25:47.013256 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011278    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 20:25:47.013256 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011281    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 20:25:47.013256 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011284    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 20:25:47.013256 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011286    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 20:25:47.013256 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011289    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 20:25:47.013256 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011292    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 20:25:47.013256 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011294    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 20:25:47.013256 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011297    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 20:25:47.013256 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011299    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 20:25:47.013256 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011302    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 20:25:47.013748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011305    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 20:25:47.013748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011307    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 20:25:47.013748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011310    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 20:25:47.013748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011314    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 20:25:47.013748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011317    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 20:25:47.013748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011320    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 20:25:47.013748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011323    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 20:25:47.013748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011325    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 20:25:47.013748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011328    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 20:25:47.013748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011331    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 20:25:47.013748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011333    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 20:25:47.013748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011336    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 20:25:47.013748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011339    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 20:25:47.013748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011341    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 20:25:47.013748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011344    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 20:25:47.013748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011346    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 20:25:47.013748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011349    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 20:25:47.013748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011352    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 20:25:47.013748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011354    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 20:25:47.013748 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011358    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 20:25:47.014263 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011360    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 20:25:47.014263 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011363    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 20:25:47.014263 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011365    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 20:25:47.014263 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011368    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 20:25:47.014263 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011370    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 20:25:47.014263 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011372    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 20:25:47.014263 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011376    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 20:25:47.014263 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011388    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 20:25:47.014263 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011391    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 20:25:47.014263 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011394    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 20:25:47.014263 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011396    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 20:25:47.014263 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011399    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 20:25:47.014263 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011401    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 20:25:47.014263 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011404    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 20:25:47.014263 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011407    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 20:25:47.014263 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011409    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 20:25:47.014263 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011412    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 20:25:47.014263 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011415    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 20:25:47.014263 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011417    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 20:25:47.014263 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011420    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 20:25:47.014806 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011423    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 20:25:47.014806 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011425    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 20:25:47.014806 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011428    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 20:25:47.014806 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011430    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 20:25:47.014806 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011433    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 20:25:47.014806 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011435    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 20:25:47.014806 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011438    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 20:25:47.014806 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011440    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 20:25:47.014806 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011443    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 20:25:47.014806 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011446    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 20:25:47.014806 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011449    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 20:25:47.014806 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011451    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 20:25:47.014806 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011454    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 20:25:47.014806 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011457    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 20:25:47.014806 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011459    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 20:25:47.014806 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011462    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 20:25:47.015193 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.011467    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 20:25:47.015193 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011578    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 20:25:47.015193 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011583    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 20:25:47.015193 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011585    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 20:25:47.015193 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011588    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 20:25:47.015193 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011591    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 20:25:47.015193 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011594    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 20:25:47.015193 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011596    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 20:25:47.015193 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011599    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 20:25:47.015193 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011601    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 20:25:47.015193 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011604    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 20:25:47.015193 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011606    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 20:25:47.015193 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011609    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 20:25:47.015193 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011611    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 20:25:47.015193 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011614    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 20:25:47.015586 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011617    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 20:25:47.015586 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011619    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 20:25:47.015586 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011621    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 20:25:47.015586 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011624    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 20:25:47.015586 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011626    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 20:25:47.015586 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011629    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 20:25:47.015586 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011631    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 20:25:47.015586 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011634    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 20:25:47.015586 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011636    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 20:25:47.015586 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011639    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 20:25:47.015586 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011642    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 20:25:47.015586 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011644    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 20:25:47.015586 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011647    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 20:25:47.015586 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011649    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 20:25:47.015586 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011652    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 20:25:47.015586 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011655    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 20:25:47.015586 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011657    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 20:25:47.015586 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011660    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 20:25:47.015586 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011663    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 20:25:47.016057 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011667    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 20:25:47.016057 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011670    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 20:25:47.016057 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011673    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 20:25:47.016057 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011676    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 20:25:47.016057 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011678    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 20:25:47.016057 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011681    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 20:25:47.016057 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011684    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 20:25:47.016057 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011686    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 20:25:47.016057 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011689    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 20:25:47.016057 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011691    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 20:25:47.016057 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011694    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 20:25:47.016057 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011696    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 20:25:47.016057 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011698    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 20:25:47.016057 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011701    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 20:25:47.016057 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011704    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 20:25:47.016057 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011706    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 20:25:47.016057 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011709    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 20:25:47.016057 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011711    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 20:25:47.016057 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011714    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 20:25:47.016557 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011717    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 20:25:47.016557 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011719    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 20:25:47.016557 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011722    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 20:25:47.016557 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011725    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 20:25:47.016557 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011727    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 20:25:47.016557 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011730    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 20:25:47.016557 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011732    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 20:25:47.016557 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011735    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 20:25:47.016557 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011737    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 20:25:47.016557 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011742    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 20:25:47.016557 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011745    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 20:25:47.016557 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011748    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 20:25:47.016557 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011751    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 20:25:47.016557 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011754    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 20:25:47.016557 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011756    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 20:25:47.016557 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011759    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 20:25:47.016557 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011762    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 20:25:47.016557 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011766    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 20:25:47.016557 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011769    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 20:25:47.017032 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011772    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 20:25:47.017032 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011775    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 20:25:47.017032 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011778    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 20:25:47.017032 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011781    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 20:25:47.017032 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011783    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 20:25:47.017032 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011787    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 20:25:47.017032 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011789    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 20:25:47.017032 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011792    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 20:25:47.017032 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011795    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 20:25:47.017032 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011797    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 20:25:47.017032 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011800    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 20:25:47.017032 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011802    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 20:25:47.017032 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011805    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 20:25:47.017032 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011808    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 20:25:47.017032 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011810    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 20:25:47.017032 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011813    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 20:25:47.017032 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011815    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 20:25:47.017032 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011818    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 20:25:47.017032 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011820    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 20:25:47.017032 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011823    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 20:25:47.017564 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011825    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 20:25:47.017564 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011828    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 20:25:47.017564 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011830    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 20:25:47.017564 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011836    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 20:25:47.017564 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011839    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 20:25:47.017564 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011842    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 20:25:47.017564 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011844    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 20:25:47.017564 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:47.011847    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 20:25:47.017564 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.011851    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 20:25:47.017564 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.011959    2552 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 20:25:47.017564 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.013734    2552 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 20:25:47.017564 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.014783    2552 server.go:1019] "Starting client certificate rotation"
Sep 08 20:25:47.017564 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.014880    2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 20:25:47.017564 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.014917    2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 20:25:47.017907 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.016965    2552 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 20:25:47.018809 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.018772    2552 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 20:25:47.028961 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.028940    2552 log.go:25] "Validated CRI v1 runtime API"
Sep 08 20:25:47.033190 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.033169    2552 log.go:25] "Validated CRI v1 image API"
Sep 08 20:25:47.034048 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.034021    2552 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 20:25:47.038330 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.038311    2552 fs.go:135] Filesystem UUIDs: map[6c20080d-cd16-4d40-951d-740c28b53be4:/dev/nvme0n1p4 6dade5e4-59df-43c7-a904-92715edc9035:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 08 20:25:47.038384 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.038329    2552 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 20:25:47.042296 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.042270    2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 20:25:47.044414 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.044296    2552 manager.go:216] Machine: {Timestamp:2026-09-08 20:25:47.042848653 +0000 UTC m=+0.294330022 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090117 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c2c145f30d0fde44f8d39d2811855 SystemUUID:ec2c2c14-5f30-d0fd-e44f-8d39d2811855 BootID:dd85b07d-989e-421e-81a5-fca482cb5102 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:20:04:32:ed:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:20:04:32:ed:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:98:cf:aa:c6:87 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 20:25:47.044414 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.044403    2552 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 20:25:47.044575 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.044512    2552 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 20:25:47.047575 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.047550    2552 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 20:25:47.047714 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.047578    2552 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-118.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 20:25:47.048403 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.048392    2552 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 20:25:47.048441 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.048405    2552 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 20:25:47.048441 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.048412    2552 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 20:25:47.048441 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.048425    2552 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 20:25:47.050070 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.050060    2552 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 20:25:47.054451 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.054439    2552 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 20:25:47.054571 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.054561    2552 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 20:25:47.054605 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.054592    2552 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 20:25:47.054633 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.054610    2552 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 20:25:47.054633 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.054624    2552 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 20:25:47.054698 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.054634    2552 kubelet.go:398] "Adding apiserver pod source"
Sep 08 20:25:47.054698 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.054643    2552 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 20:25:47.054999 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.054986    2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 20:25:47.055051 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.055007    2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 20:25:47.058629 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.058612    2552 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 20:25:47.061743 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.061729    2552 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 20:25:47.061797 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.061771    2552 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 20:25:47.061955 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.061942    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 20:25:47.061988 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.061962    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 20:25:47.061988 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.061968    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 20:25:47.061988 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.061973    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 20:25:47.061988 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.061981    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 20:25:47.061988 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.061987    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 20:25:47.062383 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.062372    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 20:25:47.062414 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.062387    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 20:25:47.062414 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.062396    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 20:25:47.062414 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.062405    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 20:25:47.062492 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.062419    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 20:25:47.062492 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.062431    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 20:25:47.063181 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:47.063158    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-118.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 20:25:47.063224 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.063211    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 20:25:47.063275 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.063229    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 20:25:47.063479 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:47.063450    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 20:25:47.065665 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.065643    2552 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-118.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 20:25:47.068282 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.068263    2552 server.go:1268] "Started kubelet"
Sep 08 20:25:47.068847 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.068818    2552 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 20:25:47.069085 ip-10-0-129-118 systemd[1]: Started Kubernetes Kubelet.
Sep 08 20:25:47.069498 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.069153    2552 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 20:25:47.070144 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.069944    2552 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 20:25:47.072188 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.072069    2552 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 20:25:47.072457 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.072426    2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tb29q"
Sep 08 20:25:47.073402 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.073384    2552 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 20:25:47.074699 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:47.074678    2552 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 20:25:47.077994 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.077982    2552 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 20:25:47.078036 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.078024    2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 20:25:47.078109 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:47.078090    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-118.ec2.internal\" not found"
Sep 08 20:25:47.078549 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.078528    2552 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 20:25:47.078645 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.078558    2552 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 20:25:47.078645 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.078569    2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tb29q"
Sep 08 20:25:47.078645 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.078574    2552 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 20:25:47.078802 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.078693    2552 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 20:25:47.078802 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.078702    2552 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 20:25:47.079340 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.079320    2552 factory.go:55] Registering systemd factory
Sep 08 20:25:47.079437 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.079344    2552 factory.go:223] Registration of the systemd container factory successfully
Sep 08 20:25:47.079608 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.079595    2552 factory.go:153] Registering CRI-O factory
Sep 08 20:25:47.079669 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.079614    2552 factory.go:223] Registration of the crio container factory successfully
Sep 08 20:25:47.079721 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.079681    2552 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 20:25:47.079721 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.079703    2552 factory.go:103] Registering Raw factory
Sep 08 20:25:47.079721 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.079718    2552 manager.go:1196] Started watching for new ooms in manager
Sep 08 20:25:47.080125 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.080110    2552 manager.go:349] Starting recovery of all containers
Sep 08 20:25:47.085573 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.085404    2552 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 20:25:47.091088 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.091053    2552 manager.go:354] Recovery completed
Sep 08 20:25:47.092574 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:47.092543    2552 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-118.ec2.internal\" not found" node="ip-10-0-129-118.ec2.internal"
Sep 08 20:25:47.098916 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.098901    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 20:25:47.101669 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.101652    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-118.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 20:25:47.101735 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.101682    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-118.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 20:25:47.101735 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.101693    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-118.ec2.internal" event="NodeHasSufficientPID"
Sep 08 20:25:47.102321 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.102305    2552 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 20:25:47.102321 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.102318    2552 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 20:25:47.102321 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.102322    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 20:25:47.102453 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.102334    2552 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 20:25:47.103025 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.103010    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-118.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 20:25:47.103099 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.103036    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-118.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 20:25:47.103099 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.103049    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-118.ec2.internal" event="NodeHasSufficientPID"
Sep 08 20:25:47.103495 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.103483    2552 policy_none.go:49] "None policy: Start"
Sep 08 20:25:47.103537 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.103501    2552 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 20:25:47.103537 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.103514    2552 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 20:25:47.104483 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.104472    2552 policy_none.go:47] "Start"
Sep 08 20:25:47.130797 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.130779    2552 manager.go:337] "Starting Device Plugin manager"
Sep 08 20:25:47.130908 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:47.130818    2552 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 20:25:47.130908 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.130831    2552 server.go:87] "Starting device plugin registration server"
Sep 08 20:25:47.131160 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.131146    2552 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 20:25:47.131398 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.131161    2552 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 20:25:47.131398 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.131314    2552 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 20:25:47.131849 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.131822    2552 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 20:25:47.131849 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.131837    2552 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 20:25:47.132399 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:47.132222    2552 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 20:25:47.132399 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:47.132312    2552 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-118.ec2.internal\" not found"
Sep 08 20:25:47.217250 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.217207    2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 20:25:47.218426 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.218410    2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 20:25:47.218492 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.218468    2552 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 20:25:47.218538 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.218493    2552 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 20:25:47.218587 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:47.218543    2552 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 20:25:47.221290 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.221269    2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 20:25:47.231568 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.231554    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 20:25:47.232453 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.232426    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-118.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 20:25:47.232543 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.232468    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-118.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 20:25:47.232543 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.232484    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-118.ec2.internal" event="NodeHasSufficientPID"
Sep 08 20:25:47.232543 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.232513    2552 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-118.ec2.internal"
Sep 08 20:25:47.241828 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.241815    2552 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-118.ec2.internal"
Sep 08 20:25:47.241886 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:47.241835    2552 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-118.ec2.internal\": node \"ip-10-0-129-118.ec2.internal\" not found"
Sep 08 20:25:47.263390 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:47.263312    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-118.ec2.internal\" not found"
Sep 08 20:25:47.318650 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.318612    2552 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-118.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-118.ec2.internal"]
Sep 08 20:25:47.318765 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.318703    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 20:25:47.320467 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.320447    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-118.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 20:25:47.320562 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.320475    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-118.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 20:25:47.320562 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.320485    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-118.ec2.internal" event="NodeHasSufficientPID"
Sep 08 20:25:47.321807 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.321795    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 20:25:47.321927 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.321914    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-118.ec2.internal"
Sep 08 20:25:47.321964 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.321942    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 20:25:47.323156 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.323141    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-118.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 20:25:47.323268 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.323165    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-118.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 20:25:47.323268 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.323177    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-118.ec2.internal" event="NodeHasSufficientPID"
Sep 08 20:25:47.323268 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.323187    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-118.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 20:25:47.323268 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.323207    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-118.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 20:25:47.323268 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.323221    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-118.ec2.internal" event="NodeHasSufficientPID"
Sep 08 20:25:47.324564 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.324548    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-118.ec2.internal"
Sep 08 20:25:47.324644 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.324573    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 20:25:47.325218 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.325203    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-118.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 20:25:47.325218 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.325227    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-118.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 20:25:47.325352 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.325252    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-118.ec2.internal" event="NodeHasSufficientPID"
Sep 08 20:25:47.350822 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:47.350800    2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-118.ec2.internal\" not found" node="ip-10-0-129-118.ec2.internal"
Sep 08 20:25:47.355107 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:47.355091    2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-118.ec2.internal\" not found" node="ip-10-0-129-118.ec2.internal"
Sep 08 20:25:47.363846 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:47.363829    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-118.ec2.internal\" not found"
Sep 08 20:25:47.379922 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.379903    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bbac504614cf2da4968012e8d77d47f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-118.ec2.internal\" (UID: \"3bbac504614cf2da4968012e8d77d47f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-118.ec2.internal"
Sep 08 20:25:47.379985 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.379925    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/791cc5370b298e2f89744b04dce519c9-config\") pod \"kube-apiserver-proxy-ip-10-0-129-118.ec2.internal\" (UID: \"791cc5370b298e2f89744b04dce519c9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-118.ec2.internal"
Sep 08 20:25:47.379985 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.379942    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3bbac504614cf2da4968012e8d77d47f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-118.ec2.internal\" (UID: \"3bbac504614cf2da4968012e8d77d47f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-118.ec2.internal"
Sep 08 20:25:47.464364 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:47.464337    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-118.ec2.internal\" not found"
Sep 08 20:25:47.480695 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.480664    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bbac504614cf2da4968012e8d77d47f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-118.ec2.internal\" (UID: \"3bbac504614cf2da4968012e8d77d47f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-118.ec2.internal"
Sep 08 20:25:47.480792 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.480719    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bbac504614cf2da4968012e8d77d47f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-118.ec2.internal\" (UID: \"3bbac504614cf2da4968012e8d77d47f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-118.ec2.internal"
Sep 08 20:25:47.480792 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.480752    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/791cc5370b298e2f89744b04dce519c9-config\") pod \"kube-apiserver-proxy-ip-10-0-129-118.ec2.internal\" (UID: \"791cc5370b298e2f89744b04dce519c9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-118.ec2.internal"
Sep 08 20:25:47.480792 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.480780    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3bbac504614cf2da4968012e8d77d47f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-118.ec2.internal\" (UID: \"3bbac504614cf2da4968012e8d77d47f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-118.ec2.internal"
Sep 08 20:25:47.480893 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.480808    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3bbac504614cf2da4968012e8d77d47f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-118.ec2.internal\" (UID: \"3bbac504614cf2da4968012e8d77d47f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-118.ec2.internal"
Sep 08 20:25:47.480893 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.480825    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/791cc5370b298e2f89744b04dce519c9-config\") pod \"kube-apiserver-proxy-ip-10-0-129-118.ec2.internal\" (UID: \"791cc5370b298e2f89744b04dce519c9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-118.ec2.internal"
Sep 08 20:25:47.565136 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:47.565069    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-118.ec2.internal\" not found"
Sep 08 20:25:47.654591 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.654551    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-118.ec2.internal"
Sep 08 20:25:47.659045 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:47.658961    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-118.ec2.internal"
Sep 08 20:25:47.666012 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:47.665993    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-118.ec2.internal\" not found"
Sep 08 20:25:47.767731 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:47.767697    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-118.ec2.internal\" not found"
Sep 08 20:25:47.868626 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:47.868550    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-118.ec2.internal\" not found"
Sep 08 20:25:47.969806 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:47.969774    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-118.ec2.internal\" not found"
Sep 08 20:25:48.009439 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:48.009413    2552 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 20:25:48.015094 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:48.015076    2552 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 20:25:48.015203 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:48.015186    2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 20:25:48.015244 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:48.015219    2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 20:25:48.015244 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:48.015228    2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 20:25:48.070824 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:48.070801    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-118.ec2.internal\" not found"
Sep 08 20:25:48.080408 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:48.080362    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 20:20:47 +0000 UTC" deadline="2028-06-05 16:36:27.077173432 +0000 UTC"
Sep 08 20:25:48.080408 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:48.080405    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15260h10m38.996772279s"
Sep 08 20:25:48.172156 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:48.172075    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-118.ec2.internal\" not found"
Sep 08 20:25:48.194741 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:48.194710    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791cc5370b298e2f89744b04dce519c9.slice/crio-3aa1d0d6ea4cf6bbf99e88558f4b38ba48d71cf1b8abf924eef8cc0c776d6479 WatchSource:0}: Error finding container 3aa1d0d6ea4cf6bbf99e88558f4b38ba48d71cf1b8abf924eef8cc0c776d6479: Status 404 returned error can't find the container with id 3aa1d0d6ea4cf6bbf99e88558f4b38ba48d71cf1b8abf924eef8cc0c776d6479
Sep 08 20:25:48.195090 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:48.195069    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bbac504614cf2da4968012e8d77d47f.slice/crio-d37eaeb3044a726e8a65b6811aad559d89654d0a8e58a811e2722129a0e8dabd WatchSource:0}: Error finding container d37eaeb3044a726e8a65b6811aad559d89654d0a8e58a811e2722129a0e8dabd: Status 404 returned error can't find the container with id d37eaeb3044a726e8a65b6811aad559d89654d0a8e58a811e2722129a0e8dabd
Sep 08 20:25:48.201670 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:48.201645    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 20:25:48.222647 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:48.222603    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-118.ec2.internal" event={"ID":"3bbac504614cf2da4968012e8d77d47f","Type":"ContainerStarted","Data":"d37eaeb3044a726e8a65b6811aad559d89654d0a8e58a811e2722129a0e8dabd"}
Sep 08 20:25:48.223729 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:48.223706    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-118.ec2.internal" event={"ID":"791cc5370b298e2f89744b04dce519c9","Type":"ContainerStarted","Data":"3aa1d0d6ea4cf6bbf99e88558f4b38ba48d71cf1b8abf924eef8cc0c776d6479"}
Sep 08 20:25:48.275841 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:48.275812    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-118.ec2.internal\" not found"
Sep 08 20:25:48.310854 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:48.310828    2552 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 20:25:48.378293 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:48.378252    2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-118.ec2.internal"
Sep 08 20:25:48.394270 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:48.394234    2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 20:25:48.395330 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:48.395304    2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-118.ec2.internal"
Sep 08 20:25:48.406898 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:48.406883    2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 20:25:49.055479 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.055434    2552 apiserver.go:52] "Watching apiserver"
Sep 08 20:25:49.062044 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.062016    2552 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 20:25:49.063578 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.063547    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-zvlkf","openshift-multus/network-metrics-daemon-zwvng","openshift-ovn-kubernetes/ovnkube-node-5kdmb","kube-system/konnectivity-agent-xfczt","kube-system/kube-apiserver-proxy-ip-10-0-129-118.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v","openshift-cluster-node-tuning-operator/tuned-svl8z","openshift-network-diagnostics/network-check-target-g8mj2","openshift-network-operator/iptables-alerter-8fmh6","kube-system/global-pull-secret-syncer-f5v7c","openshift-dns/node-resolver-th7xh","openshift-image-registry/node-ca-zxmp4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-118.ec2.internal","openshift-multus/multus-9fmtz"]
Sep 08 20:25:49.065273 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.065233    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.066544 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.066497    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-th7xh"
Sep 08 20:25:49.067843 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.067785    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-j5d7h\""
Sep 08 20:25:49.067843 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.067804    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 20:25:49.068036 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.067972    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 20:25:49.068136 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.068120    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.069497 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.069476    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 20:25:49.069690 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.069664    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-t2w4f\""
Sep 08 20:25:49.069899 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.069883    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 20:25:49.070339 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.070209    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xfczt"
Sep 08 20:25:49.071402 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.070994    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zvlkf"
Sep 08 20:25:49.072410 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.072390    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v"
Sep 08 20:25:49.072685 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.072664    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 20:25:49.072830 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.072811    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 20:25:49.073742 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.073717    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 20:25:49.073889 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.073860    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 20:25:49.073997 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.073940    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zhrfm\""
Sep 08 20:25:49.074337 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.074318    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8fmh6"
Sep 08 20:25:49.075780 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.075761    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.075906 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.075842    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 20:25:49.076080 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.076058    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8nls4\""
Sep 08 20:25:49.076193 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.076101    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dg2vt\""
Sep 08 20:25:49.076193 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.076173    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 20:25:49.076355 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.075843    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 20:25:49.076440 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.076390    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 20:25:49.076863 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.076540    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 20:25:49.076863 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.076549    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 20:25:49.076863 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.076672    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 20:25:49.077145 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.076881    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 20:25:49.077612 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.077185    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5xwfw\""
Sep 08 20:25:49.078413 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.078389    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:25:49.078672 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:49.078483    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwvng" podUID="0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059"
Sep 08 20:25:49.083201 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.081337    2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 20:25:49.083201 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.081980    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 20:25:49.083201 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.082170    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hjnw4\""
Sep 08 20:25:49.083201 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.082448    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 20:25:49.083201 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.082454    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 20:25:49.083201 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.082514    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 20:25:49.083201 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.082712    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 20:25:49.083201 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.082814    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:25:49.083201 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.082940    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 20:25:49.083201 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.082973    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7vdsn\""
Sep 08 20:25:49.083201 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:49.082974    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8mj2" podUID="cb9818b0-76cd-4be1-9c9e-7d47e0cb1607"
Sep 08 20:25:49.083201 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.083079    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 20:25:49.083840 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.083393    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 20:25:49.083840 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.083592    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zxmp4"
Sep 08 20:25:49.085735 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.085713    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:25:49.085837 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:49.085787    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5v7c" podUID="0dc52e01-2095-40dd-8603-0fdf9d6ff1aa"
Sep 08 20:25:49.086886 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.086353    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 20:25:49.086886 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.086701    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-59rt8\""
Sep 08 20:25:49.086886 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.086751    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 20:25:49.087100 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.086928    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 20:25:49.090904 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.090876    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-host-kubelet\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.091011 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.090912    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-run-systemd\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.091011 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.090941    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/135336c8-5355-4a06-bd24-30cf54fd420a-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvlkf\" (UID: \"135336c8-5355-4a06-bd24-30cf54fd420a\") " pod="openshift-multus/multus-additional-cni-plugins-zvlkf"
Sep 08 20:25:49.091011 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.090965    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-etc-modprobe-d\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.091011 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.090986    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-etc-sysconfig\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.091011 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091001    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db8d6722-08c4-48b2-aade-b5e405d482fa-host-slash\") pod \"iptables-alerter-8fmh6\" (UID: \"db8d6722-08c4-48b2-aade-b5e405d482fa\") " pod="openshift-network-operator/iptables-alerter-8fmh6"
Sep 08 20:25:49.091299 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091027    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bced89cc-7b97-493f-b71d-8fc32bc7187f-hosts-file\") pod \"node-resolver-th7xh\" (UID: \"bced89cc-7b97-493f-b71d-8fc32bc7187f\") " pod="openshift-dns/node-resolver-th7xh"
Sep 08 20:25:49.091299 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091044    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-systemd-units\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.091299 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091061    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-env-overrides\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.091299 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091075    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/135336c8-5355-4a06-bd24-30cf54fd420a-cnibin\") pod \"multus-additional-cni-plugins-zvlkf\" (UID: \"135336c8-5355-4a06-bd24-30cf54fd420a\") " pod="openshift-multus/multus-additional-cni-plugins-zvlkf"
Sep 08 20:25:49.091299 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091089    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-etc-tuned\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.091299 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091118    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-tmp\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.091299 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091147    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0fd7b065-bbbb-458a-840d-da97ea47930a-multus-daemon-config\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.091299 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091172    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-log-socket\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.091299 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091216    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/135336c8-5355-4a06-bd24-30cf54fd420a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvlkf\" (UID: \"135336c8-5355-4a06-bd24-30cf54fd420a\") " pod="openshift-multus/multus-additional-cni-plugins-zvlkf"
Sep 08 20:25:49.091299 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091254    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/584911f8-e71c-4c68-a868-95949a7bfaa9-sys-fs\") pod \"aws-ebs-csi-driver-node-br99v\" (UID: \"584911f8-e71c-4c68-a868-95949a7bfaa9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v"
Sep 08 20:25:49.091299 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091282    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-host-var-lib-cni-bin\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.091907 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091332    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6mhf\" (UniqueName: \"kubernetes.io/projected/0fd7b065-bbbb-458a-840d-da97ea47930a-kube-api-access-d6mhf\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.091907 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091380    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/135336c8-5355-4a06-bd24-30cf54fd420a-system-cni-dir\") pod \"multus-additional-cni-plugins-zvlkf\" (UID: \"135336c8-5355-4a06-bd24-30cf54fd420a\") " pod="openshift-multus/multus-additional-cni-plugins-zvlkf"
Sep 08 20:25:49.091907 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091409    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/584911f8-e71c-4c68-a868-95949a7bfaa9-device-dir\") pod \"aws-ebs-csi-driver-node-br99v\" (UID: \"584911f8-e71c-4c68-a868-95949a7bfaa9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v"
Sep 08 20:25:49.091907 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091462    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-host-cni-bin\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.091907 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091519    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-host-cni-netd\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.091907 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091557    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/135336c8-5355-4a06-bd24-30cf54fd420a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvlkf\" (UID: \"135336c8-5355-4a06-bd24-30cf54fd420a\") " pod="openshift-multus/multus-additional-cni-plugins-zvlkf"
Sep 08 20:25:49.091907 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091635    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtsj2\" (UniqueName: \"kubernetes.io/projected/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-kube-api-access-rtsj2\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.091907 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091795    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-host-var-lib-cni-multus\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.091907 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091829    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-var-lib-openvswitch\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.091907 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091863    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db8d6722-08c4-48b2-aade-b5e405d482fa-iptables-alerter-script\") pod \"iptables-alerter-8fmh6\" (UID: \"db8d6722-08c4-48b2-aade-b5e405d482fa\") " pod="openshift-network-operator/iptables-alerter-8fmh6"
Sep 08 20:25:49.091907 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091886    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-host-var-lib-kubelet\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.091907 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091902    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-host-run-multus-certs\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.092534 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091953    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-node-log\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.092534 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.091985    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.092534 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092011    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-var-lib-kubelet\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.092534 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092059    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-etc-openvswitch\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.092534 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092119    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d616a62-5fd9-40e1-b415-a4cd644a83e8-konnectivity-ca\") pod \"konnectivity-agent-xfczt\" (UID: \"6d616a62-5fd9-40e1-b415-a4cd644a83e8\") " pod="kube-system/konnectivity-agent-xfczt"
Sep 08 20:25:49.092534 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092145    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzchv\" (UniqueName: \"kubernetes.io/projected/135336c8-5355-4a06-bd24-30cf54fd420a-kube-api-access-bzchv\") pod \"multus-additional-cni-plugins-zvlkf\" (UID: \"135336c8-5355-4a06-bd24-30cf54fd420a\") " pod="openshift-multus/multus-additional-cni-plugins-zvlkf"
Sep 08 20:25:49.092534 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092217    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6ttj\" (UniqueName: \"kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj\") pod \"network-check-target-g8mj2\" (UID: \"cb9818b0-76cd-4be1-9c9e-7d47e0cb1607\") " pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:25:49.092534 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092269    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-sys\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.092534 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092351    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-host\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.092534 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092384    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-host-slash\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.092534 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092425    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/584911f8-e71c-4c68-a868-95949a7bfaa9-registration-dir\") pod \"aws-ebs-csi-driver-node-br99v\" (UID: \"584911f8-e71c-4c68-a868-95949a7bfaa9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v"
Sep 08 20:25:49.092534 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092452    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xltx9\" (UniqueName: \"kubernetes.io/projected/584911f8-e71c-4c68-a868-95949a7bfaa9-kube-api-access-xltx9\") pod \"aws-ebs-csi-driver-node-br99v\" (UID: \"584911f8-e71c-4c68-a868-95949a7bfaa9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v"
Sep 08 20:25:49.092534 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092487    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-multus-socket-dir-parent\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.092534 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092534    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-etc-kubernetes\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.093236 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092562    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-hostroot\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.093236 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092654    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-system-cni-dir\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.093236 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092692    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bced89cc-7b97-493f-b71d-8fc32bc7187f-tmp-dir\") pod \"node-resolver-th7xh\" (UID: \"bced89cc-7b97-493f-b71d-8fc32bc7187f\") " pod="openshift-dns/node-resolver-th7xh"
Sep 08 20:25:49.093236 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092741    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-run-openvswitch\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.093236 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092769    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-host-run-ovn-kubernetes\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.093236 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092809    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-ovn-node-metrics-cert\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.093236 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092844    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-etc-sysctl-d\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.093236 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092870    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-run\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.093236 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092896    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-ovnkube-config\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.093236 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092921    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d616a62-5fd9-40e1-b415-a4cd644a83e8-agent-certs\") pod \"konnectivity-agent-xfczt\" (UID: \"6d616a62-5fd9-40e1-b415-a4cd644a83e8\") " pod="kube-system/konnectivity-agent-xfczt"
Sep 08 20:25:49.093236 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092955    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w7tj\" (UniqueName: \"kubernetes.io/projected/db8d6722-08c4-48b2-aade-b5e405d482fa-kube-api-access-4w7tj\") pod \"iptables-alerter-8fmh6\" (UID: \"db8d6722-08c4-48b2-aade-b5e405d482fa\") " pod="openshift-network-operator/iptables-alerter-8fmh6"
Sep 08 20:25:49.093236 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.092985    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-multus-cni-dir\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.093236 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.093027    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-cnibin\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.093236 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.093051    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-host-run-k8s-cni-cncf-io\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.093236 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.093077    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-multus-conf-dir\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.093236 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.093120    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jch8w\" (UniqueName: \"kubernetes.io/projected/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-kube-api-access-jch8w\") pod \"network-metrics-daemon-zwvng\" (UID: \"0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059\") " pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:25:49.093236 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.093160    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-run-ovn\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.094219 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.093186    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/584911f8-e71c-4c68-a868-95949a7bfaa9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-br99v\" (UID: \"584911f8-e71c-4c68-a868-95949a7bfaa9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v"
Sep 08 20:25:49.094219 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.093224    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-etc-sysctl-conf\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.094219 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.093271    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-lib-modules\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.094219 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.093296    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0fd7b065-bbbb-458a-840d-da97ea47930a-cni-binary-copy\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.094219 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.093320    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-host-run-netns\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.094219 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.093372    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/135336c8-5355-4a06-bd24-30cf54fd420a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvlkf\" (UID: \"135336c8-5355-4a06-bd24-30cf54fd420a\") " pod="openshift-multus/multus-additional-cni-plugins-zvlkf"
Sep 08 20:25:49.094219 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.093410    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/584911f8-e71c-4c68-a868-95949a7bfaa9-etc-selinux\") pod \"aws-ebs-csi-driver-node-br99v\" (UID: \"584911f8-e71c-4c68-a868-95949a7bfaa9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v"
Sep 08 20:25:49.094219 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.093437    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-etc-systemd\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.094219 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.093466    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-ovnkube-script-lib\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.094219 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.093492    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lddhh\" (UniqueName: \"kubernetes.io/projected/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-kube-api-access-lddhh\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.094219 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.093514    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-host-run-netns\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.094219 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.093539    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45hvl\" (UniqueName: \"kubernetes.io/projected/bced89cc-7b97-493f-b71d-8fc32bc7187f-kube-api-access-45hvl\") pod \"node-resolver-th7xh\" (UID: \"bced89cc-7b97-493f-b71d-8fc32bc7187f\") " pod="openshift-dns/node-resolver-th7xh"
Sep 08 20:25:49.094219 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.093602    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/584911f8-e71c-4c68-a868-95949a7bfaa9-socket-dir\") pod \"aws-ebs-csi-driver-node-br99v\" (UID: \"584911f8-e71c-4c68-a868-95949a7bfaa9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v"
Sep 08 20:25:49.094219 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.093641    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-etc-kubernetes\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.094219 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.093674    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs\") pod \"network-metrics-daemon-zwvng\" (UID: \"0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059\") " pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:25:49.097255 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.097213    2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 20:25:49.106472 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.106448    2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 20:25:49.127368 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.127320    2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fvf22"
Sep 08 20:25:49.136436 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.136413    2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fvf22"
Sep 08 20:25:49.180400 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.180365    2552 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 20:25:49.194765 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.194731    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-multus-socket-dir-parent\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.194916 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.194776    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-etc-kubernetes\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.194916 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.194891    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-multus-socket-dir-parent\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.194916 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.194901    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-hostroot\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.195087 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.194945    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-kubelet-config\") pod \"global-pull-secret-syncer-f5v7c\" (UID: \"0dc52e01-2095-40dd-8603-0fdf9d6ff1aa\") " pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:25:49.195087 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.194954    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-hostroot\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.195087 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.194980    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-system-cni-dir\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.195087 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195008    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bced89cc-7b97-493f-b71d-8fc32bc7187f-tmp-dir\") pod \"node-resolver-th7xh\" (UID: \"bced89cc-7b97-493f-b71d-8fc32bc7187f\") " pod="openshift-dns/node-resolver-th7xh"
Sep 08 20:25:49.195087 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195031    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-etc-kubernetes\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.195087 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195035    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-run-openvswitch\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.195087 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195074    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-host-run-ovn-kubernetes\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.195415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195100    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-ovn-node-metrics-cert\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.195415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195106    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-system-cni-dir\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.195415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195123    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/345de38f-94c6-4067-a5c6-fc7ad6a560e9-host\") pod \"node-ca-zxmp4\" (UID: \"345de38f-94c6-4067-a5c6-fc7ad6a560e9\") " pod="openshift-image-registry/node-ca-zxmp4"
Sep 08 20:25:49.195415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195159    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/345de38f-94c6-4067-a5c6-fc7ad6a560e9-serviceca\") pod \"node-ca-zxmp4\" (UID: \"345de38f-94c6-4067-a5c6-fc7ad6a560e9\") " pod="openshift-image-registry/node-ca-zxmp4"
Sep 08 20:25:49.195415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195183    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-etc-sysctl-d\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.195415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195204    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-run-openvswitch\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.195415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195208    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-run\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.195415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195239    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-ovnkube-config\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.195415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195282    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d616a62-5fd9-40e1-b415-a4cd644a83e8-agent-certs\") pod \"konnectivity-agent-xfczt\" (UID: \"6d616a62-5fd9-40e1-b415-a4cd644a83e8\") " pod="kube-system/konnectivity-agent-xfczt"
Sep 08 20:25:49.195415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195307    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w7tj\" (UniqueName: \"kubernetes.io/projected/db8d6722-08c4-48b2-aade-b5e405d482fa-kube-api-access-4w7tj\") pod \"iptables-alerter-8fmh6\" (UID: \"db8d6722-08c4-48b2-aade-b5e405d482fa\") " pod="openshift-network-operator/iptables-alerter-8fmh6"
Sep 08 20:25:49.195415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195334    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-multus-cni-dir\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.195415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195359    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-cnibin\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.195415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195370    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-etc-sysctl-d\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.195415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195387    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-host-run-k8s-cni-cncf-io\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.195415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195157    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-host-run-ovn-kubernetes\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.195415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195413    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-multus-conf-dir\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.196207 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195438    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jch8w\" (UniqueName: \"kubernetes.io/projected/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-kube-api-access-jch8w\") pod \"network-metrics-daemon-zwvng\" (UID: \"0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059\") " pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:25:49.196207 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195466    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-run-ovn\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.196207 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195491    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/584911f8-e71c-4c68-a868-95949a7bfaa9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-br99v\" (UID: \"584911f8-e71c-4c68-a868-95949a7bfaa9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v"
Sep 08 20:25:49.196207 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195530    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-etc-sysctl-conf\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.196207 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195553    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-lib-modules\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.196207 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195580    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0fd7b065-bbbb-458a-840d-da97ea47930a-cni-binary-copy\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.196207 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195611    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-host-run-netns\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.196207 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195629    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-multus-conf-dir\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.196207 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195641    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/135336c8-5355-4a06-bd24-30cf54fd420a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvlkf\" (UID: \"135336c8-5355-4a06-bd24-30cf54fd420a\") " pod="openshift-multus/multus-additional-cni-plugins-zvlkf"
Sep 08 20:25:49.196207 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195669    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/584911f8-e71c-4c68-a868-95949a7bfaa9-etc-selinux\") pod \"aws-ebs-csi-driver-node-br99v\" (UID: \"584911f8-e71c-4c68-a868-95949a7bfaa9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v"
Sep 08 20:25:49.196207 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195666    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/584911f8-e71c-4c68-a868-95949a7bfaa9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-br99v\" (UID: \"584911f8-e71c-4c68-a868-95949a7bfaa9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v"
Sep 08 20:25:49.196207 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195698    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-etc-systemd\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.196207 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195712    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-run-ovn\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.196207 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195725    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-ovnkube-script-lib\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.196207 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195762    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lddhh\" (UniqueName: \"kubernetes.io/projected/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-kube-api-access-lddhh\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.196207 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195886    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-etc-systemd\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.196207 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195961    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-multus-cni-dir\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.197133 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.195963    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/584911f8-e71c-4c68-a868-95949a7bfaa9-etc-selinux\") pod \"aws-ebs-csi-driver-node-br99v\" (UID: \"584911f8-e71c-4c68-a868-95949a7bfaa9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v"
Sep 08 20:25:49.197133 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.196019    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-host-run-netns\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.197133 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.196027    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-host-run-k8s-cni-cncf-io\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.197133 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.196074    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-cnibin\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.197133 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.196209    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-etc-sysctl-conf\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.197133 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.196214    2552 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 20:25:49.197133 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.196331    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0fd7b065-bbbb-458a-840d-da97ea47930a-cni-binary-copy\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.197133 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.196388    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/135336c8-5355-4a06-bd24-30cf54fd420a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvlkf\" (UID: \"135336c8-5355-4a06-bd24-30cf54fd420a\") " pod="openshift-multus/multus-additional-cni-plugins-zvlkf"
Sep 08 20:25:49.197133 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.196441    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-host-run-netns\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.197133 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.196467    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-ovnkube-script-lib\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.197133 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.196474    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45hvl\" (UniqueName: \"kubernetes.io/projected/bced89cc-7b97-493f-b71d-8fc32bc7187f-kube-api-access-45hvl\") pod \"node-resolver-th7xh\" (UID: \"bced89cc-7b97-493f-b71d-8fc32bc7187f\") " pod="openshift-dns/node-resolver-th7xh"
Sep 08 20:25:49.197133 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.196510    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/584911f8-e71c-4c68-a868-95949a7bfaa9-socket-dir\") pod \"aws-ebs-csi-driver-node-br99v\" (UID: \"584911f8-e71c-4c68-a868-95949a7bfaa9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v"
Sep 08 20:25:49.197133 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.196535    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-host-run-netns\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.197133 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.196539    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-etc-kubernetes\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.197133 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.196584    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-etc-kubernetes\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.197133 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.196599    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs\") pod \"network-metrics-daemon-zwvng\" (UID: \"0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059\") " pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:25:49.197133 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.196625    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-host-kubelet\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.197133 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.196647    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-lib-modules\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.197975 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:49.196744    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 20:25:49.197975 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.196766    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/584911f8-e71c-4c68-a868-95949a7bfaa9-socket-dir\") pod \"aws-ebs-csi-driver-node-br99v\" (UID: \"584911f8-e71c-4c68-a868-95949a7bfaa9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v"
Sep 08 20:25:49.197975 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.196653    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-run-systemd\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.197975 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.196743    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-ovnkube-config\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.197975 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:49.196818    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs podName:0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059 nodeName:}" failed. No retries permitted until 2026-09-08 20:25:49.696797201 +0000 UTC m=+2.948278539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs") pod "network-metrics-daemon-zwvng" (UID: "0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 20:25:49.197975 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.196844    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-host-kubelet\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.197975 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.196817    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-run-systemd\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.197975 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.196875    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/135336c8-5355-4a06-bd24-30cf54fd420a-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvlkf\" (UID: \"135336c8-5355-4a06-bd24-30cf54fd420a\") " pod="openshift-multus/multus-additional-cni-plugins-zvlkf"
Sep 08 20:25:49.197975 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.197214    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bced89cc-7b97-493f-b71d-8fc32bc7187f-tmp-dir\") pod \"node-resolver-th7xh\" (UID: \"bced89cc-7b97-493f-b71d-8fc32bc7187f\") " pod="openshift-dns/node-resolver-th7xh"
Sep 08 20:25:49.197975 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.197532    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-etc-modprobe-d\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.197975 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.197579    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-etc-sysconfig\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.197975 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.197613    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db8d6722-08c4-48b2-aade-b5e405d482fa-host-slash\") pod \"iptables-alerter-8fmh6\" (UID: \"db8d6722-08c4-48b2-aade-b5e405d482fa\") " pod="openshift-network-operator/iptables-alerter-8fmh6"
Sep 08 20:25:49.197975 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.197665    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bced89cc-7b97-493f-b71d-8fc32bc7187f-hosts-file\") pod \"node-resolver-th7xh\" (UID: \"bced89cc-7b97-493f-b71d-8fc32bc7187f\") " pod="openshift-dns/node-resolver-th7xh"
Sep 08 20:25:49.197975 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.197685    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-etc-modprobe-d\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.197975 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.197697    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-systemd-units\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.197975 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.197710    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-run\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.197975 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.197723    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-env-overrides\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.198887 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.197826    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/135336c8-5355-4a06-bd24-30cf54fd420a-cnibin\") pod \"multus-additional-cni-plugins-zvlkf\" (UID: \"135336c8-5355-4a06-bd24-30cf54fd420a\") " pod="openshift-multus/multus-additional-cni-plugins-zvlkf"
Sep 08 20:25:49.198887 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.197855    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krm7h\" (UniqueName: \"kubernetes.io/projected/345de38f-94c6-4067-a5c6-fc7ad6a560e9-kube-api-access-krm7h\") pod \"node-ca-zxmp4\" (UID: \"345de38f-94c6-4067-a5c6-fc7ad6a560e9\") " pod="openshift-image-registry/node-ca-zxmp4"
Sep 08 20:25:49.198887 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.197887    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-etc-tuned\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.198887 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.197898    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/135336c8-5355-4a06-bd24-30cf54fd420a-cnibin\") pod \"multus-additional-cni-plugins-zvlkf\" (UID: \"135336c8-5355-4a06-bd24-30cf54fd420a\") " pod="openshift-multus/multus-additional-cni-plugins-zvlkf"
Sep 08 20:25:49.198887 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.197912    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-tmp\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.198887 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.197941    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-systemd-units\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.198887 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.198023    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-etc-sysconfig\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.198887 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.198059    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db8d6722-08c4-48b2-aade-b5e405d482fa-host-slash\") pod \"iptables-alerter-8fmh6\" (UID: \"db8d6722-08c4-48b2-aade-b5e405d482fa\") " pod="openshift-network-operator/iptables-alerter-8fmh6"
Sep 08 20:25:49.198887 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.198450    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/135336c8-5355-4a06-bd24-30cf54fd420a-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvlkf\" (UID: \"135336c8-5355-4a06-bd24-30cf54fd420a\") " pod="openshift-multus/multus-additional-cni-plugins-zvlkf"
Sep 08 20:25:49.198887 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.198485    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-env-overrides\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.198887 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.198540    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bced89cc-7b97-493f-b71d-8fc32bc7187f-hosts-file\") pod \"node-resolver-th7xh\" (UID: \"bced89cc-7b97-493f-b71d-8fc32bc7187f\") " pod="openshift-dns/node-resolver-th7xh"
Sep 08 20:25:49.198887 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.198588    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0fd7b065-bbbb-458a-840d-da97ea47930a-multus-daemon-config\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.198887 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.198634    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-log-socket\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.198887 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.198661    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/135336c8-5355-4a06-bd24-30cf54fd420a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvlkf\" (UID: \"135336c8-5355-4a06-bd24-30cf54fd420a\") " pod="openshift-multus/multus-additional-cni-plugins-zvlkf"
Sep 08 20:25:49.198887 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.198713    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/584911f8-e71c-4c68-a868-95949a7bfaa9-sys-fs\") pod \"aws-ebs-csi-driver-node-br99v\" (UID: \"584911f8-e71c-4c68-a868-95949a7bfaa9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v"
Sep 08 20:25:49.198887 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.198739    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-dbus\") pod \"global-pull-secret-syncer-f5v7c\" (UID: \"0dc52e01-2095-40dd-8603-0fdf9d6ff1aa\") " pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:25:49.198887 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.198765    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-host-var-lib-cni-bin\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.199749 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.198792    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6mhf\" (UniqueName: \"kubernetes.io/projected/0fd7b065-bbbb-458a-840d-da97ea47930a-kube-api-access-d6mhf\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.199749 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.198853    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/135336c8-5355-4a06-bd24-30cf54fd420a-system-cni-dir\") pod \"multus-additional-cni-plugins-zvlkf\" (UID: \"135336c8-5355-4a06-bd24-30cf54fd420a\") " pod="openshift-multus/multus-additional-cni-plugins-zvlkf"
Sep 08 20:25:49.199749 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.198879    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/584911f8-e71c-4c68-a868-95949a7bfaa9-device-dir\") pod \"aws-ebs-csi-driver-node-br99v\" (UID: \"584911f8-e71c-4c68-a868-95949a7bfaa9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v"
Sep 08 20:25:49.199749 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.198902    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-host-cni-bin\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.199749 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.198925    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-host-cni-netd\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.199749 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.198950    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/135336c8-5355-4a06-bd24-30cf54fd420a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvlkf\" (UID: \"135336c8-5355-4a06-bd24-30cf54fd420a\") " pod="openshift-multus/multus-additional-cni-plugins-zvlkf"
Sep 08 20:25:49.199749 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.198978    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtsj2\" (UniqueName: \"kubernetes.io/projected/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-kube-api-access-rtsj2\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.199749 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.199002    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-host-var-lib-cni-multus\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.199749 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.199028    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-var-lib-openvswitch\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.199749 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.199112    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db8d6722-08c4-48b2-aade-b5e405d482fa-iptables-alerter-script\") pod \"iptables-alerter-8fmh6\" (UID: \"db8d6722-08c4-48b2-aade-b5e405d482fa\") " pod="openshift-network-operator/iptables-alerter-8fmh6"
Sep 08 20:25:49.199749 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.199135    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-host-var-lib-kubelet\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.199749 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.199159    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-host-run-multus-certs\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.199749 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.199181    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-node-log\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.199749 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.199206    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.199749 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.199230    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-var-lib-kubelet\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.199749 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.199255    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-etc-openvswitch\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.199749 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.199280    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d616a62-5fd9-40e1-b415-a4cd644a83e8-konnectivity-ca\") pod \"konnectivity-agent-xfczt\" (UID: \"6d616a62-5fd9-40e1-b415-a4cd644a83e8\") " pod="kube-system/konnectivity-agent-xfczt"
Sep 08 20:25:49.201030 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.199322    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzchv\" (UniqueName: \"kubernetes.io/projected/135336c8-5355-4a06-bd24-30cf54fd420a-kube-api-access-bzchv\") pod \"multus-additional-cni-plugins-zvlkf\" (UID: \"135336c8-5355-4a06-bd24-30cf54fd420a\") " pod="openshift-multus/multus-additional-cni-plugins-zvlkf"
Sep 08 20:25:49.201030 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.199356    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ttj\" (UniqueName: \"kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj\") pod \"network-check-target-g8mj2\" (UID: \"cb9818b0-76cd-4be1-9c9e-7d47e0cb1607\") " pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:25:49.201030 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.199378    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-sys\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.201030 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.199399    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-host\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.201030 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.199423    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-host-slash\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.201030 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.199446    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/584911f8-e71c-4c68-a868-95949a7bfaa9-registration-dir\") pod \"aws-ebs-csi-driver-node-br99v\" (UID: \"584911f8-e71c-4c68-a868-95949a7bfaa9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v"
Sep 08 20:25:49.201030 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.199476    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xltx9\" (UniqueName: \"kubernetes.io/projected/584911f8-e71c-4c68-a868-95949a7bfaa9-kube-api-access-xltx9\") pod \"aws-ebs-csi-driver-node-br99v\" (UID: \"584911f8-e71c-4c68-a868-95949a7bfaa9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v"
Sep 08 20:25:49.201030 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.199638    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret\") pod \"global-pull-secret-syncer-f5v7c\" (UID: \"0dc52e01-2095-40dd-8603-0fdf9d6ff1aa\") " pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:25:49.201030 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.199915    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-var-lib-openvswitch\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.201030 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.199927    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-host-var-lib-cni-bin\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.201030 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.199940    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-host-var-lib-cni-multus\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.201030 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.199969    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-host-run-multus-certs\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.201030 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.199987    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/135336c8-5355-4a06-bd24-30cf54fd420a-system-cni-dir\") pod \"multus-additional-cni-plugins-zvlkf\" (UID: \"135336c8-5355-4a06-bd24-30cf54fd420a\") " pod="openshift-multus/multus-additional-cni-plugins-zvlkf"
Sep 08 20:25:49.201030 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.200005    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-node-log\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.201030 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.200029    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/584911f8-e71c-4c68-a868-95949a7bfaa9-device-dir\") pod \"aws-ebs-csi-driver-node-br99v\" (UID: \"584911f8-e71c-4c68-a868-95949a7bfaa9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v"
Sep 08 20:25:49.201030 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.200062    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-host-cni-bin\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.201030 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.200066    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.202069 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.200093    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-host-cni-netd\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.202069 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.200117    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-var-lib-kubelet\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.202069 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.200151    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-etc-openvswitch\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.202069 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.200501    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db8d6722-08c4-48b2-aade-b5e405d482fa-iptables-alerter-script\") pod \"iptables-alerter-8fmh6\" (UID: \"db8d6722-08c4-48b2-aade-b5e405d482fa\") " pod="openshift-network-operator/iptables-alerter-8fmh6"
Sep 08 20:25:49.202069 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.200588    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-log-socket\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.202069 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.200606    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/135336c8-5355-4a06-bd24-30cf54fd420a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvlkf\" (UID: \"135336c8-5355-4a06-bd24-30cf54fd420a\") " pod="openshift-multus/multus-additional-cni-plugins-zvlkf"
Sep 08 20:25:49.202069 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.200687    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-etc-tuned\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.202069 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.200746    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/584911f8-e71c-4c68-a868-95949a7bfaa9-sys-fs\") pod \"aws-ebs-csi-driver-node-br99v\" (UID: \"584911f8-e71c-4c68-a868-95949a7bfaa9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v"
Sep 08 20:25:49.202069 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.200752    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/135336c8-5355-4a06-bd24-30cf54fd420a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvlkf\" (UID: \"135336c8-5355-4a06-bd24-30cf54fd420a\") " pod="openshift-multus/multus-additional-cni-plugins-zvlkf"
Sep 08 20:25:49.202069 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.200765    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fd7b065-bbbb-458a-840d-da97ea47930a-host-var-lib-kubelet\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.202069 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.200822    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-sys\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.202069 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.200858    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-host-slash\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.202069 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.200909    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-host\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.202069 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.200949    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/584911f8-e71c-4c68-a868-95949a7bfaa9-registration-dir\") pod \"aws-ebs-csi-driver-node-br99v\" (UID: \"584911f8-e71c-4c68-a868-95949a7bfaa9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v"
Sep 08 20:25:49.202069 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.201143    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d616a62-5fd9-40e1-b415-a4cd644a83e8-agent-certs\") pod \"konnectivity-agent-xfczt\" (UID: \"6d616a62-5fd9-40e1-b415-a4cd644a83e8\") " pod="kube-system/konnectivity-agent-xfczt"
Sep 08 20:25:49.202069 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.201202    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-tmp\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.202069 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.201214    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0fd7b065-bbbb-458a-840d-da97ea47930a-multus-daemon-config\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.202069 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.201536    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d616a62-5fd9-40e1-b415-a4cd644a83e8-konnectivity-ca\") pod \"konnectivity-agent-xfczt\" (UID: \"6d616a62-5fd9-40e1-b415-a4cd644a83e8\") " pod="kube-system/konnectivity-agent-xfczt"
Sep 08 20:25:49.203392 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.201805    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-ovn-node-metrics-cert\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.207621 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.207003    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lddhh\" (UniqueName: \"kubernetes.io/projected/4f3b5c49-0e9f-4179-bd36-b019c23a08b7-kube-api-access-lddhh\") pod \"ovnkube-node-5kdmb\" (UID: \"4f3b5c49-0e9f-4179-bd36-b019c23a08b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.207621 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.207568    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jch8w\" (UniqueName: \"kubernetes.io/projected/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-kube-api-access-jch8w\") pod \"network-metrics-daemon-zwvng\" (UID: \"0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059\") " pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:25:49.214272 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.211681    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w7tj\" (UniqueName: \"kubernetes.io/projected/db8d6722-08c4-48b2-aade-b5e405d482fa-kube-api-access-4w7tj\") pod \"iptables-alerter-8fmh6\" (UID: \"db8d6722-08c4-48b2-aade-b5e405d482fa\") " pod="openshift-network-operator/iptables-alerter-8fmh6"
Sep 08 20:25:49.214272 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:49.212827    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 20:25:49.214272 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:49.212856    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 20:25:49.214272 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:49.212871    2552 projected.go:196] Error preparing data for projected volume kube-api-access-x6ttj for pod openshift-network-diagnostics/network-check-target-g8mj2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 20:25:49.214272 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:49.212940    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj podName:cb9818b0-76cd-4be1-9c9e-7d47e0cb1607 nodeName:}" failed. No retries permitted until 2026-09-08 20:25:49.712920751 +0000 UTC m=+2.964402103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x6ttj" (UniqueName: "kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj") pod "network-check-target-g8mj2" (UID: "cb9818b0-76cd-4be1-9c9e-7d47e0cb1607") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 20:25:49.215919 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.215894    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xltx9\" (UniqueName: \"kubernetes.io/projected/584911f8-e71c-4c68-a868-95949a7bfaa9-kube-api-access-xltx9\") pod \"aws-ebs-csi-driver-node-br99v\" (UID: \"584911f8-e71c-4c68-a868-95949a7bfaa9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v"
Sep 08 20:25:49.216474 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.216428    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzchv\" (UniqueName: \"kubernetes.io/projected/135336c8-5355-4a06-bd24-30cf54fd420a-kube-api-access-bzchv\") pod \"multus-additional-cni-plugins-zvlkf\" (UID: \"135336c8-5355-4a06-bd24-30cf54fd420a\") " pod="openshift-multus/multus-additional-cni-plugins-zvlkf"
Sep 08 20:25:49.216590 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.216566    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45hvl\" (UniqueName: \"kubernetes.io/projected/bced89cc-7b97-493f-b71d-8fc32bc7187f-kube-api-access-45hvl\") pod \"node-resolver-th7xh\" (UID: \"bced89cc-7b97-493f-b71d-8fc32bc7187f\") " pod="openshift-dns/node-resolver-th7xh"
Sep 08 20:25:49.217787 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.217698    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6mhf\" (UniqueName: \"kubernetes.io/projected/0fd7b065-bbbb-458a-840d-da97ea47930a-kube-api-access-d6mhf\") pod \"multus-9fmtz\" (UID: \"0fd7b065-bbbb-458a-840d-da97ea47930a\") " pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.220590 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.220568    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtsj2\" (UniqueName: \"kubernetes.io/projected/5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab-kube-api-access-rtsj2\") pod \"tuned-svl8z\" (UID: \"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab\") " pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.300747 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.300707    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krm7h\" (UniqueName: \"kubernetes.io/projected/345de38f-94c6-4067-a5c6-fc7ad6a560e9-kube-api-access-krm7h\") pod \"node-ca-zxmp4\" (UID: \"345de38f-94c6-4067-a5c6-fc7ad6a560e9\") " pod="openshift-image-registry/node-ca-zxmp4"
Sep 08 20:25:49.300747 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.300747    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-dbus\") pod \"global-pull-secret-syncer-f5v7c\" (UID: \"0dc52e01-2095-40dd-8603-0fdf9d6ff1aa\") " pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:25:49.300948 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.300786    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret\") pod \"global-pull-secret-syncer-f5v7c\" (UID: \"0dc52e01-2095-40dd-8603-0fdf9d6ff1aa\") " pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:25:49.300948 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.300811    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-kubelet-config\") pod \"global-pull-secret-syncer-f5v7c\" (UID: \"0dc52e01-2095-40dd-8603-0fdf9d6ff1aa\") " pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:25:49.300948 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.300864    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-dbus\") pod \"global-pull-secret-syncer-f5v7c\" (UID: \"0dc52e01-2095-40dd-8603-0fdf9d6ff1aa\") " pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:25:49.300948 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.300870    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-kubelet-config\") pod \"global-pull-secret-syncer-f5v7c\" (UID: \"0dc52e01-2095-40dd-8603-0fdf9d6ff1aa\") " pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:25:49.301161 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:49.300984    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 20:25:49.301161 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.300989    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/345de38f-94c6-4067-a5c6-fc7ad6a560e9-host\") pod \"node-ca-zxmp4\" (UID: \"345de38f-94c6-4067-a5c6-fc7ad6a560e9\") " pod="openshift-image-registry/node-ca-zxmp4"
Sep 08 20:25:49.301161 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.301032    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/345de38f-94c6-4067-a5c6-fc7ad6a560e9-serviceca\") pod \"node-ca-zxmp4\" (UID: \"345de38f-94c6-4067-a5c6-fc7ad6a560e9\") " pod="openshift-image-registry/node-ca-zxmp4"
Sep 08 20:25:49.301161 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:49.301050    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret podName:0dc52e01-2095-40dd-8603-0fdf9d6ff1aa nodeName:}" failed. No retries permitted until 2026-09-08 20:25:49.801030758 +0000 UTC m=+3.052512107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret") pod "global-pull-secret-syncer-f5v7c" (UID: "0dc52e01-2095-40dd-8603-0fdf9d6ff1aa") : object "kube-system"/"original-pull-secret" not registered
Sep 08 20:25:49.301161 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.301080    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/345de38f-94c6-4067-a5c6-fc7ad6a560e9-host\") pod \"node-ca-zxmp4\" (UID: \"345de38f-94c6-4067-a5c6-fc7ad6a560e9\") " pod="openshift-image-registry/node-ca-zxmp4"
Sep 08 20:25:49.302160 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.302141    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/345de38f-94c6-4067-a5c6-fc7ad6a560e9-serviceca\") pod \"node-ca-zxmp4\" (UID: \"345de38f-94c6-4067-a5c6-fc7ad6a560e9\") " pod="openshift-image-registry/node-ca-zxmp4"
Sep 08 20:25:49.312360 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.312303    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krm7h\" (UniqueName: \"kubernetes.io/projected/345de38f-94c6-4067-a5c6-fc7ad6a560e9-kube-api-access-krm7h\") pod \"node-ca-zxmp4\" (UID: \"345de38f-94c6-4067-a5c6-fc7ad6a560e9\") " pod="openshift-image-registry/node-ca-zxmp4"
Sep 08 20:25:49.387327 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.387294    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-svl8z"
Sep 08 20:25:49.397421 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.397399    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-th7xh"
Sep 08 20:25:49.408470 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.408445    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:25:49.413215 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.413190    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xfczt"
Sep 08 20:25:49.419880 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.419855    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zvlkf"
Sep 08 20:25:49.426489 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.426471    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v"
Sep 08 20:25:49.434198 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.434176    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8fmh6"
Sep 08 20:25:49.440984 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.440966    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9fmtz"
Sep 08 20:25:49.447584 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.447568    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zxmp4"
Sep 08 20:25:49.475836 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.475812    2552 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 20:25:49.703795 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.703711    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs\") pod \"network-metrics-daemon-zwvng\" (UID: \"0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059\") " pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:25:49.703957 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:49.703856    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 20:25:49.703957 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:49.703936    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs podName:0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059 nodeName:}" failed. No retries permitted until 2026-09-08 20:25:50.703919471 +0000 UTC m=+3.955400813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs") pod "network-metrics-daemon-zwvng" (UID: "0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 20:25:49.804965 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.804920    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ttj\" (UniqueName: \"kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj\") pod \"network-check-target-g8mj2\" (UID: \"cb9818b0-76cd-4be1-9c9e-7d47e0cb1607\") " pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:25:49.805164 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:49.804980    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret\") pod \"global-pull-secret-syncer-f5v7c\" (UID: \"0dc52e01-2095-40dd-8603-0fdf9d6ff1aa\") " pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:25:49.805164 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:49.805108    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 20:25:49.805164 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:49.805125    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 20:25:49.805164 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:49.805148    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 20:25:49.805164 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:49.805165    2552 projected.go:196] Error preparing data for projected volume kube-api-access-x6ttj for pod openshift-network-diagnostics/network-check-target-g8mj2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 20:25:49.805428 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:49.805189    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret podName:0dc52e01-2095-40dd-8603-0fdf9d6ff1aa nodeName:}" failed. No retries permitted until 2026-09-08 20:25:50.805166678 +0000 UTC m=+4.056648020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret") pod "global-pull-secret-syncer-f5v7c" (UID: "0dc52e01-2095-40dd-8603-0fdf9d6ff1aa") : object "kube-system"/"original-pull-secret" not registered
Sep 08 20:25:49.805428 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:49.805218    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj podName:cb9818b0-76cd-4be1-9c9e-7d47e0cb1607 nodeName:}" failed. No retries permitted until 2026-09-08 20:25:50.805205309 +0000 UTC m=+4.056686646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x6ttj" (UniqueName: "kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj") pod "network-check-target-g8mj2" (UID: "cb9818b0-76cd-4be1-9c9e-7d47e0cb1607") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 20:25:50.139304 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:50.139195    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 20:20:49 +0000 UTC" deadline="2028-05-29 19:48:46.346054746 +0000 UTC"
Sep 08 20:25:50.139304 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:50.139237    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15095h22m56.206822399s"
Sep 08 20:25:50.219331 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:50.219297    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:25:50.219481 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:50.219401    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8mj2" podUID="cb9818b0-76cd-4be1-9c9e-7d47e0cb1607"
Sep 08 20:25:50.261481 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:50.261322    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2ecfe6_4d86_40bb_81e3_31b01d11b5ab.slice/crio-f72bcc181888553a88236514108d48ab770a50ae3827ab7156a1aae0a3e647af WatchSource:0}: Error finding container f72bcc181888553a88236514108d48ab770a50ae3827ab7156a1aae0a3e647af: Status 404 returned error can't find the container with id f72bcc181888553a88236514108d48ab770a50ae3827ab7156a1aae0a3e647af
Sep 08 20:25:50.262975 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:50.262812    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d616a62_5fd9_40e1_b415_a4cd644a83e8.slice/crio-50c2f097a1d33d9c6e1e169db344b9737c26382865e5ad2c41ab8c4e413ed00d WatchSource:0}: Error finding container 50c2f097a1d33d9c6e1e169db344b9737c26382865e5ad2c41ab8c4e413ed00d: Status 404 returned error can't find the container with id 50c2f097a1d33d9c6e1e169db344b9737c26382865e5ad2c41ab8c4e413ed00d
Sep 08 20:25:50.264846 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:50.264820    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f3b5c49_0e9f_4179_bd36_b019c23a08b7.slice/crio-c71c28dc03e81580890abb406a8002ceb8914dac1c14448ebfa5702c27a114a0 WatchSource:0}: Error finding container c71c28dc03e81580890abb406a8002ceb8914dac1c14448ebfa5702c27a114a0: Status 404 returned error can't find the container with id c71c28dc03e81580890abb406a8002ceb8914dac1c14448ebfa5702c27a114a0
Sep 08 20:25:50.266980 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:50.266954    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135336c8_5355_4a06_bd24_30cf54fd420a.slice/crio-996517ce69886c2f6e85f3ca50630fc22bb2808978505d22357388eb7718449a WatchSource:0}: Error finding container 996517ce69886c2f6e85f3ca50630fc22bb2808978505d22357388eb7718449a: Status 404 returned error can't find the container with id 996517ce69886c2f6e85f3ca50630fc22bb2808978505d22357388eb7718449a
Sep 08 20:25:50.269733 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:50.269704    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbced89cc_7b97_493f_b71d_8fc32bc7187f.slice/crio-4397d93f311e3213e1984b9dcdf3dc22a57e9a43a8b91d4744389dde49cc5872 WatchSource:0}: Error finding container 4397d93f311e3213e1984b9dcdf3dc22a57e9a43a8b91d4744389dde49cc5872: Status 404 returned error can't find the container with id 4397d93f311e3213e1984b9dcdf3dc22a57e9a43a8b91d4744389dde49cc5872
Sep 08 20:25:50.271341 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:50.271276    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8d6722_08c4_48b2_aade_b5e405d482fa.slice/crio-680128018d2aa923037d4a029d66e721beb7ea9135171315262fddf4597397d6 WatchSource:0}: Error finding container 680128018d2aa923037d4a029d66e721beb7ea9135171315262fddf4597397d6: Status 404 returned error can't find the container with id 680128018d2aa923037d4a029d66e721beb7ea9135171315262fddf4597397d6
Sep 08 20:25:50.273366 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:50.273348    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd7b065_bbbb_458a_840d_da97ea47930a.slice/crio-b1faf14884b604371cbcd91912aa76eed206ddd404bad541df74e624e1cb31fa WatchSource:0}: Error finding container b1faf14884b604371cbcd91912aa76eed206ddd404bad541df74e624e1cb31fa: Status 404 returned error can't find the container with id b1faf14884b604371cbcd91912aa76eed206ddd404bad541df74e624e1cb31fa
Sep 08 20:25:50.276271 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:50.276234    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod345de38f_94c6_4067_a5c6_fc7ad6a560e9.slice/crio-80584c9b83a419f0fccf21007b806e081b2bf5b0dbd12bff39aa6f72d8a5102a WatchSource:0}: Error finding container 80584c9b83a419f0fccf21007b806e081b2bf5b0dbd12bff39aa6f72d8a5102a: Status 404 returned error can't find the container with id 80584c9b83a419f0fccf21007b806e081b2bf5b0dbd12bff39aa6f72d8a5102a
Sep 08 20:25:50.276827 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:25:50.276738    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584911f8_e71c_4c68_a868_95949a7bfaa9.slice/crio-5414cd67f5b5b8015218adff611bea379050f97e01859a830598a1c1c1f53231 WatchSource:0}: Error finding container 5414cd67f5b5b8015218adff611bea379050f97e01859a830598a1c1c1f53231: Status 404 returned error can't find the container with id 5414cd67f5b5b8015218adff611bea379050f97e01859a830598a1c1c1f53231
Sep 08 20:25:50.714875 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:50.714836    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs\") pod \"network-metrics-daemon-zwvng\" (UID: \"0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059\") " pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:25:50.715023 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:50.714959    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 20:25:50.715023 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:50.715017    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs podName:0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059 nodeName:}" failed. No retries permitted until 2026-09-08 20:25:52.715003177 +0000 UTC m=+5.966484513 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs") pod "network-metrics-daemon-zwvng" (UID: "0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 20:25:50.815255 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:50.815163    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ttj\" (UniqueName: \"kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj\") pod \"network-check-target-g8mj2\" (UID: \"cb9818b0-76cd-4be1-9c9e-7d47e0cb1607\") " pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:25:50.815255 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:50.815230    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret\") pod \"global-pull-secret-syncer-f5v7c\" (UID: \"0dc52e01-2095-40dd-8603-0fdf9d6ff1aa\") " pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:25:50.815523 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:50.815447    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 20:25:50.815523 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:50.815521    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret podName:0dc52e01-2095-40dd-8603-0fdf9d6ff1aa nodeName:}" failed. No retries permitted until 2026-09-08 20:25:52.815499883 +0000 UTC m=+6.066981242 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret") pod "global-pull-secret-syncer-f5v7c" (UID: "0dc52e01-2095-40dd-8603-0fdf9d6ff1aa") : object "kube-system"/"original-pull-secret" not registered
Sep 08 20:25:50.816125 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:50.815959    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 20:25:50.816125 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:50.815985    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 20:25:50.816125 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:50.815999    2552 projected.go:196] Error preparing data for projected volume kube-api-access-x6ttj for pod openshift-network-diagnostics/network-check-target-g8mj2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 20:25:50.816125 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:50.816052    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj podName:cb9818b0-76cd-4be1-9c9e-7d47e0cb1607 nodeName:}" failed. No retries permitted until 2026-09-08 20:25:52.816035742 +0000 UTC m=+6.067517080 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x6ttj" (UniqueName: "kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj") pod "network-check-target-g8mj2" (UID: "cb9818b0-76cd-4be1-9c9e-7d47e0cb1607") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 20:25:51.142016 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:51.141885    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 20:20:49 +0000 UTC" deadline="2028-04-26 05:28:31.968967453 +0000 UTC"
Sep 08 20:25:51.142016 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:51.141953    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14289h2m40.827019058s"
Sep 08 20:25:51.170839 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:51.170495    2552 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 20:25:51.222859 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:51.221814    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:25:51.222859 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:51.221974    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwvng" podUID="0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059"
Sep 08 20:25:51.222859 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:51.222693    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:25:51.222859 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:51.222798    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5v7c" podUID="0dc52e01-2095-40dd-8603-0fdf9d6ff1aa"
Sep 08 20:25:51.248993 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:51.248561    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-118.ec2.internal" event={"ID":"791cc5370b298e2f89744b04dce519c9","Type":"ContainerStarted","Data":"c7d40dc614cf8115dcc32f4c5cbb5eaaa33e28d2e31b3b6b3daa8183012d64f1"}
Sep 08 20:25:51.259887 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:51.259306    2552 generic.go:356] "Generic (PLEG): container finished" podID="3bbac504614cf2da4968012e8d77d47f" containerID="c05e7ce2fb0dd3493f2572f97d827a5b367e58c1176c8c5932cc47c8e4cbe3ff" exitCode=0
Sep 08 20:25:51.259887 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:51.259425    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-118.ec2.internal" event={"ID":"3bbac504614cf2da4968012e8d77d47f","Type":"ContainerDied","Data":"c05e7ce2fb0dd3493f2572f97d827a5b367e58c1176c8c5932cc47c8e4cbe3ff"}
Sep 08 20:25:51.265409 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:51.263393    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v" event={"ID":"584911f8-e71c-4c68-a868-95949a7bfaa9","Type":"ContainerStarted","Data":"5414cd67f5b5b8015218adff611bea379050f97e01859a830598a1c1c1f53231"}
Sep 08 20:25:51.280462 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:51.280022    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-118.ec2.internal" podStartSLOduration=3.28000105 podStartE2EDuration="3.28000105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 20:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:25:51.265195798 +0000 UTC m=+4.516677160" watchObservedRunningTime="2026-09-08 20:25:51.28000105 +0000 UTC m=+4.531482409"
Sep 08 20:25:51.306494 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:51.306400    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8fmh6" event={"ID":"db8d6722-08c4-48b2-aade-b5e405d482fa","Type":"ContainerStarted","Data":"680128018d2aa923037d4a029d66e721beb7ea9135171315262fddf4597397d6"}
Sep 08 20:25:51.318456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:51.318426    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb" event={"ID":"4f3b5c49-0e9f-4179-bd36-b019c23a08b7","Type":"ContainerStarted","Data":"c71c28dc03e81580890abb406a8002ceb8914dac1c14448ebfa5702c27a114a0"}
Sep 08 20:25:51.330387 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:51.330350    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvlkf" event={"ID":"135336c8-5355-4a06-bd24-30cf54fd420a","Type":"ContainerStarted","Data":"996517ce69886c2f6e85f3ca50630fc22bb2808978505d22357388eb7718449a"}
Sep 08 20:25:51.336863 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:51.336807    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xfczt" event={"ID":"6d616a62-5fd9-40e1-b415-a4cd644a83e8","Type":"ContainerStarted","Data":"50c2f097a1d33d9c6e1e169db344b9737c26382865e5ad2c41ab8c4e413ed00d"}
Sep 08 20:25:51.354101 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:51.354070    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zxmp4" event={"ID":"345de38f-94c6-4067-a5c6-fc7ad6a560e9","Type":"ContainerStarted","Data":"80584c9b83a419f0fccf21007b806e081b2bf5b0dbd12bff39aa6f72d8a5102a"}
Sep 08 20:25:51.363548 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:51.363487    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9fmtz" event={"ID":"0fd7b065-bbbb-458a-840d-da97ea47930a","Type":"ContainerStarted","Data":"b1faf14884b604371cbcd91912aa76eed206ddd404bad541df74e624e1cb31fa"}
Sep 08 20:25:51.376466 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:51.376379    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-th7xh" event={"ID":"bced89cc-7b97-493f-b71d-8fc32bc7187f","Type":"ContainerStarted","Data":"4397d93f311e3213e1984b9dcdf3dc22a57e9a43a8b91d4744389dde49cc5872"}
Sep 08 20:25:51.393649 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:51.393562    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-svl8z" event={"ID":"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab","Type":"ContainerStarted","Data":"f72bcc181888553a88236514108d48ab770a50ae3827ab7156a1aae0a3e647af"}
Sep 08 20:25:52.219570 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:52.219484    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:25:52.220184 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:52.219629    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8mj2" podUID="cb9818b0-76cd-4be1-9c9e-7d47e0cb1607"
Sep 08 20:25:52.412452 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:52.412377    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-118.ec2.internal" event={"ID":"3bbac504614cf2da4968012e8d77d47f","Type":"ContainerStarted","Data":"f94b61481d59a9d594ff1e2e455133a907bea2a73bf0d3a0d02e3634784957a7"}
Sep 08 20:25:52.732377 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:52.732339    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs\") pod \"network-metrics-daemon-zwvng\" (UID: \"0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059\") " pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:25:52.732587 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:52.732517    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 20:25:52.732587 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:52.732588    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs podName:0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059 nodeName:}" failed. No retries permitted until 2026-09-08 20:25:56.732567056 +0000 UTC m=+9.984048400 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs") pod "network-metrics-daemon-zwvng" (UID: "0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 20:25:52.834202 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:52.833367    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ttj\" (UniqueName: \"kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj\") pod \"network-check-target-g8mj2\" (UID: \"cb9818b0-76cd-4be1-9c9e-7d47e0cb1607\") " pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:25:52.834202 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:52.833423    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret\") pod \"global-pull-secret-syncer-f5v7c\" (UID: \"0dc52e01-2095-40dd-8603-0fdf9d6ff1aa\") " pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:25:52.834202 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:52.833579    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 20:25:52.834202 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:52.833650    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret podName:0dc52e01-2095-40dd-8603-0fdf9d6ff1aa nodeName:}" failed. No retries permitted until 2026-09-08 20:25:56.833627445 +0000 UTC m=+10.085108784 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret") pod "global-pull-secret-syncer-f5v7c" (UID: "0dc52e01-2095-40dd-8603-0fdf9d6ff1aa") : object "kube-system"/"original-pull-secret" not registered
Sep 08 20:25:52.834202 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:52.834080    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 20:25:52.834202 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:52.834099    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 20:25:52.834202 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:52.834117    2552 projected.go:196] Error preparing data for projected volume kube-api-access-x6ttj for pod openshift-network-diagnostics/network-check-target-g8mj2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 20:25:52.834202 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:52.834163    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj podName:cb9818b0-76cd-4be1-9c9e-7d47e0cb1607 nodeName:}" failed. No retries permitted until 2026-09-08 20:25:56.834148012 +0000 UTC m=+10.085629352 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x6ttj" (UniqueName: "kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj") pod "network-check-target-g8mj2" (UID: "cb9818b0-76cd-4be1-9c9e-7d47e0cb1607") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 20:25:53.222176 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:53.221952    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:25:53.222176 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:53.222105    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwvng" podUID="0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059"
Sep 08 20:25:53.222739 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:53.222530    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:25:53.226362 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:53.223042    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5v7c" podUID="0dc52e01-2095-40dd-8603-0fdf9d6ff1aa"
Sep 08 20:25:54.219131 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:54.219097    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:25:54.219413 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:54.219388    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8mj2" podUID="cb9818b0-76cd-4be1-9c9e-7d47e0cb1607"
Sep 08 20:25:55.220383 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:55.220354    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:25:55.220832 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:55.220486    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwvng" podUID="0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059"
Sep 08 20:25:55.220900 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:55.220363    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:25:55.221019 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:55.220983    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5v7c" podUID="0dc52e01-2095-40dd-8603-0fdf9d6ff1aa"
Sep 08 20:25:56.219556 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:56.219475    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:25:56.219730 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:56.219613    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8mj2" podUID="cb9818b0-76cd-4be1-9c9e-7d47e0cb1607"
Sep 08 20:25:56.771371 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:56.771333    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs\") pod \"network-metrics-daemon-zwvng\" (UID: \"0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059\") " pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:25:56.771840 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:56.771528    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 20:25:56.771840 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:56.771597    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs podName:0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059 nodeName:}" failed. No retries permitted until 2026-09-08 20:26:04.771577267 +0000 UTC m=+18.023058605 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs") pod "network-metrics-daemon-zwvng" (UID: "0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 20:25:56.872443 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:56.872412    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ttj\" (UniqueName: \"kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj\") pod \"network-check-target-g8mj2\" (UID: \"cb9818b0-76cd-4be1-9c9e-7d47e0cb1607\") " pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:25:56.872627 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:56.872458    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret\") pod \"global-pull-secret-syncer-f5v7c\" (UID: \"0dc52e01-2095-40dd-8603-0fdf9d6ff1aa\") " pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:25:56.872627 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:56.872612    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 20:25:56.872754 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:56.872634    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 20:25:56.872754 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:56.872661    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 20:25:56.872754 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:56.872675    2552 projected.go:196] Error preparing data for projected volume kube-api-access-x6ttj for pod openshift-network-diagnostics/network-check-target-g8mj2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 20:25:56.872754 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:56.872677    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret podName:0dc52e01-2095-40dd-8603-0fdf9d6ff1aa nodeName:}" failed. No retries permitted until 2026-09-08 20:26:04.872659045 +0000 UTC m=+18.124140382 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret") pod "global-pull-secret-syncer-f5v7c" (UID: "0dc52e01-2095-40dd-8603-0fdf9d6ff1aa") : object "kube-system"/"original-pull-secret" not registered
Sep 08 20:25:56.872754 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:56.872727    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj podName:cb9818b0-76cd-4be1-9c9e-7d47e0cb1607 nodeName:}" failed. No retries permitted until 2026-09-08 20:26:04.872709696 +0000 UTC m=+18.124191057 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x6ttj" (UniqueName: "kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj") pod "network-check-target-g8mj2" (UID: "cb9818b0-76cd-4be1-9c9e-7d47e0cb1607") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 20:25:57.220933 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:57.219844    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:25:57.220933 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:57.219974    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwvng" podUID="0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059"
Sep 08 20:25:57.220933 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:57.220023    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:25:57.220933 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:57.220136    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5v7c" podUID="0dc52e01-2095-40dd-8603-0fdf9d6ff1aa"
Sep 08 20:25:58.219453 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:58.219398    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:25:58.219929 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:58.219532    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8mj2" podUID="cb9818b0-76cd-4be1-9c9e-7d47e0cb1607"
Sep 08 20:25:59.218848 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:59.218815    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:25:59.219001 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:59.218946    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwvng" podUID="0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059"
Sep 08 20:25:59.219089 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:25:59.219007    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:25:59.219144 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:25:59.219121    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5v7c" podUID="0dc52e01-2095-40dd-8603-0fdf9d6ff1aa"
Sep 08 20:26:00.219435 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:00.219259    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:26:00.219832 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:00.219520    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8mj2" podUID="cb9818b0-76cd-4be1-9c9e-7d47e0cb1607"
Sep 08 20:26:01.219662 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:01.219629    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:26:01.220132 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:01.219773    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5v7c" podUID="0dc52e01-2095-40dd-8603-0fdf9d6ff1aa"
Sep 08 20:26:01.230098 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:01.230073    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:26:01.230273 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:01.230197    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwvng" podUID="0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059"
Sep 08 20:26:02.219265 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:02.219222    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:26:02.219435 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:02.219371    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8mj2" podUID="cb9818b0-76cd-4be1-9c9e-7d47e0cb1607"
Sep 08 20:26:03.219087 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:03.219041    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:26:03.219563 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:03.219042    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:26:03.219563 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:03.219211    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5v7c" podUID="0dc52e01-2095-40dd-8603-0fdf9d6ff1aa"
Sep 08 20:26:03.219563 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:03.219251    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwvng" podUID="0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059"
Sep 08 20:26:04.219287 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:04.219257    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:26:04.219664 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:04.219356    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8mj2" podUID="cb9818b0-76cd-4be1-9c9e-7d47e0cb1607"
Sep 08 20:26:04.836226 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:04.836193    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs\") pod \"network-metrics-daemon-zwvng\" (UID: \"0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059\") " pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:26:04.836397 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:04.836364    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 20:26:04.836453 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:04.836443    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs podName:0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059 nodeName:}" failed. No retries permitted until 2026-09-08 20:26:20.836421162 +0000 UTC m=+34.087902516 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs") pod "network-metrics-daemon-zwvng" (UID: "0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 20:26:04.937379 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:04.937345    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ttj\" (UniqueName: \"kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj\") pod \"network-check-target-g8mj2\" (UID: \"cb9818b0-76cd-4be1-9c9e-7d47e0cb1607\") " pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:26:04.937379 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:04.937384    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret\") pod \"global-pull-secret-syncer-f5v7c\" (UID: \"0dc52e01-2095-40dd-8603-0fdf9d6ff1aa\") " pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:26:04.937609 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:04.937497    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 20:26:04.937609 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:04.937540    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 20:26:04.937609 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:04.937556    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret podName:0dc52e01-2095-40dd-8603-0fdf9d6ff1aa nodeName:}" failed. No retries permitted until 2026-09-08 20:26:20.937541344 +0000 UTC m=+34.189022679 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret") pod "global-pull-secret-syncer-f5v7c" (UID: "0dc52e01-2095-40dd-8603-0fdf9d6ff1aa") : object "kube-system"/"original-pull-secret" not registered
Sep 08 20:26:04.937609 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:04.937567    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 20:26:04.937609 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:04.937581    2552 projected.go:196] Error preparing data for projected volume kube-api-access-x6ttj for pod openshift-network-diagnostics/network-check-target-g8mj2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 20:26:04.937807 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:04.937634    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj podName:cb9818b0-76cd-4be1-9c9e-7d47e0cb1607 nodeName:}" failed. No retries permitted until 2026-09-08 20:26:20.937620488 +0000 UTC m=+34.189101839 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x6ttj" (UniqueName: "kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj") pod "network-check-target-g8mj2" (UID: "cb9818b0-76cd-4be1-9c9e-7d47e0cb1607") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 20:26:05.219342 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:05.219268    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:26:05.219342 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:05.219314    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:26:05.219787 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:05.219410    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwvng" podUID="0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059"
Sep 08 20:26:05.219787 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:05.219559    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5v7c" podUID="0dc52e01-2095-40dd-8603-0fdf9d6ff1aa"
Sep 08 20:26:06.218898 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:06.218857    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:26:06.219061 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:06.218981    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8mj2" podUID="cb9818b0-76cd-4be1-9c9e-7d47e0cb1607"
Sep 08 20:26:07.220151 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:07.220114    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:26:07.220632 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:07.220253    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5v7c" podUID="0dc52e01-2095-40dd-8603-0fdf9d6ff1aa"
Sep 08 20:26:07.220632 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:07.220275    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:26:07.220632 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:07.220407    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwvng" podUID="0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059"
Sep 08 20:26:08.219522 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:08.219486    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:26:08.219714 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:08.219594    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8mj2" podUID="cb9818b0-76cd-4be1-9c9e-7d47e0cb1607"
Sep 08 20:26:09.219449 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:09.219199    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:26:09.220084 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:09.219199    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:26:09.220084 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:09.219548    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwvng" podUID="0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059"
Sep 08 20:26:09.220084 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:09.219601    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5v7c" podUID="0dc52e01-2095-40dd-8603-0fdf9d6ff1aa"
Sep 08 20:26:09.454419 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:09.454373    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb" event={"ID":"4f3b5c49-0e9f-4179-bd36-b019c23a08b7","Type":"ContainerStarted","Data":"a6789860e4c291899b4f0e4eed22e8a48dd2c138b2407d74944e6025e3b77885"}
Sep 08 20:26:09.454636 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:09.454461    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb" event={"ID":"4f3b5c49-0e9f-4179-bd36-b019c23a08b7","Type":"ContainerStarted","Data":"99029f43f7ce490e6e2e1e2b43a9d5b5196eed19cd545c52e8176e074e9ef9dd"}
Sep 08 20:26:09.454636 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:09.454508    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb" event={"ID":"4f3b5c49-0e9f-4179-bd36-b019c23a08b7","Type":"ContainerStarted","Data":"cb46466f37ed7e2d873a0eaec8fbea11340a90f40490f59655637dee79843ba8"}
Sep 08 20:26:09.454636 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:09.454554    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb" event={"ID":"4f3b5c49-0e9f-4179-bd36-b019c23a08b7","Type":"ContainerStarted","Data":"10870eded931a33a92c98aa2ea523a317ddbd6cb09ec2d6fba54ea9052249437"}
Sep 08 20:26:09.454636 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:09.454636    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb" event={"ID":"4f3b5c49-0e9f-4179-bd36-b019c23a08b7","Type":"ContainerStarted","Data":"a5fb8a5957b0e1a7ae361ce76fa755ccf02d3020aec37046366246e825898f60"}
Sep 08 20:26:09.454843 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:09.454706    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb" event={"ID":"4f3b5c49-0e9f-4179-bd36-b019c23a08b7","Type":"ContainerStarted","Data":"9625add9235443c0c45006b026bd9cb71ca5d40b3e9bee5e76c2d61bc4530d7f"}
Sep 08 20:26:09.456325 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:09.456293    2552 generic.go:356] "Generic (PLEG): container finished" podID="135336c8-5355-4a06-bd24-30cf54fd420a" containerID="0d3c073ad943e1fdd22aeb795635d9dae4968f8204b11dad4335fa5653708c73" exitCode=0
Sep 08 20:26:09.456453 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:09.456381    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvlkf" event={"ID":"135336c8-5355-4a06-bd24-30cf54fd420a","Type":"ContainerDied","Data":"0d3c073ad943e1fdd22aeb795635d9dae4968f8204b11dad4335fa5653708c73"}
Sep 08 20:26:09.458343 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:09.458310    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xfczt" event={"ID":"6d616a62-5fd9-40e1-b415-a4cd644a83e8","Type":"ContainerStarted","Data":"51787e498ad470622a69b0eb9245593c4f861fbc0a96d9ad6fd36cca6b63a2bc"}
Sep 08 20:26:09.460089 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:09.460064    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zxmp4" event={"ID":"345de38f-94c6-4067-a5c6-fc7ad6a560e9","Type":"ContainerStarted","Data":"fba5bb25e20e0793226babbca106c5cbe497c37d93a9fc51efbcb23e82d0dee6"}
Sep 08 20:26:09.461712 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:09.461664    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9fmtz" event={"ID":"0fd7b065-bbbb-458a-840d-da97ea47930a","Type":"ContainerStarted","Data":"17705e83844b0dc04a36a61fa6c27010445a9422bfbb1b6d965d0f4f4e1fc1dd"}
Sep 08 20:26:09.463603 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:09.463577    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-th7xh" event={"ID":"bced89cc-7b97-493f-b71d-8fc32bc7187f","Type":"ContainerStarted","Data":"62bd0e819bd6a9b0af78cc981b075fc71696cebfa4a47f78b5ba31916948e96b"}
Sep 08 20:26:09.465498 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:09.465474    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-svl8z" event={"ID":"5b2ecfe6-4d86-40bb-81e3-31b01d11b5ab","Type":"ContainerStarted","Data":"3740fb3219c0ee721d1a9a16710f1eb55cb8b8ae18d8d87adf8a71c5749beac1"}
Sep 08 20:26:09.468132 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:09.468104    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v" event={"ID":"584911f8-e71c-4c68-a868-95949a7bfaa9","Type":"ContainerStarted","Data":"45ab6946779198ca197f8d245e720e939ce7dfc07ffdfdc29f9242720b5ef2b9"}
Sep 08 20:26:09.479189 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:09.479150    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-118.ec2.internal" podStartSLOduration=21.479138465 podStartE2EDuration="21.479138465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 20:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:25:52.431134602 +0000 UTC m=+5.682615964" watchObservedRunningTime="2026-09-08 20:26:09.479138465 +0000 UTC m=+22.730619822"
Sep 08 20:26:09.498376 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:09.498331    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9fmtz" podStartSLOduration=4.09507708 podStartE2EDuration="22.498321257s" totalImagesPullingTime="18.403244177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 20:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:26:09.498191827 +0000 UTC m=+22.749673184" watchObservedRunningTime="2026-09-08 20:26:09.498321257 +0000 UTC m=+22.749802611"
Sep 08 20:26:09.514818 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:09.514753    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-svl8z" podStartSLOduration=4.272335797 podStartE2EDuration="22.51474197s" totalImagesPullingTime="18.242406173s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 20:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:26:09.514556828 +0000 UTC m=+22.766038196" watchObservedRunningTime="2026-09-08 20:26:09.51474197 +0000 UTC m=+22.766223326"
Sep 08 20:26:09.529479 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:09.529440    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xfczt" podStartSLOduration=4.365244224 podStartE2EDuration="22.529425152s" totalImagesPullingTime="18.164180928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:26:09.529070335 +0000 UTC m=+22.780551692" watchObservedRunningTime="2026-09-08 20:26:09.529425152 +0000 UTC m=+22.780906511"
Sep 08 20:26:09.544689 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:09.544638    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zxmp4" podStartSLOduration=3.318245602 podStartE2EDuration="21.544622962s" totalImagesPullingTime="18.22637736s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 20:25:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:26:09.544487146 +0000 UTC m=+22.795968505" watchObservedRunningTime="2026-09-08 20:26:09.544622962 +0000 UTC m=+22.796104322"
Sep 08 20:26:09.560116 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:09.560068    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-th7xh" podStartSLOduration=4.325833633 podStartE2EDuration="22.560049617s" totalImagesPullingTime="18.234215984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 20:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:26:09.559467314 +0000 UTC m=+22.810948671" watchObservedRunningTime="2026-09-08 20:26:09.560049617 +0000 UTC m=+22.811530973"
Sep 08 20:26:10.218917 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:10.218886    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:26:10.219078 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:10.219015    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8mj2" podUID="cb9818b0-76cd-4be1-9c9e-7d47e0cb1607"
Sep 08 20:26:10.321149 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:10.321122    2552 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 20:26:10.474363 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:10.474257    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8fmh6" event={"ID":"db8d6722-08c4-48b2-aade-b5e405d482fa","Type":"ContainerStarted","Data":"76c65db59729016eb58adf19af3da5f9e8941eca499cbad5194fc03240448bdc"}
Sep 08 20:26:10.476399 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:10.476370    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v" event={"ID":"584911f8-e71c-4c68-a868-95949a7bfaa9","Type":"ContainerStarted","Data":"64e902cb406cb97255e495a8058e423adbd78841fe33597f9dcf39b29c4388d5"}
Sep 08 20:26:11.145192 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:11.145071    2552 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T20:26:10.321143954Z","UUID":"6821a864-c304-4af2-93b8-2c43fbedd29b","Handler":null,"Name":"","Endpoint":""}
Sep 08 20:26:11.146963 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:11.146935    2552 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 20:26:11.147101 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:11.146976    2552 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 20:26:11.219101 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:11.219076    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:26:11.219297 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:11.219107    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:26:11.219297 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:11.219200    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwvng" podUID="0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059"
Sep 08 20:26:11.219426 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:11.219323    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5v7c" podUID="0dc52e01-2095-40dd-8603-0fdf9d6ff1aa"
Sep 08 20:26:11.482436 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:11.482406    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb" event={"ID":"4f3b5c49-0e9f-4179-bd36-b019c23a08b7","Type":"ContainerStarted","Data":"9288af77eb5a01b39efc33a1e21b49c1845bedc900d7e458d01ad87668507e01"}
Sep 08 20:26:11.546999 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:11.546968    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xfczt"
Sep 08 20:26:12.219177 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:12.219144    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:26:12.219391 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:12.219309    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8mj2" podUID="cb9818b0-76cd-4be1-9c9e-7d47e0cb1607"
Sep 08 20:26:12.486209 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:12.486127    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v" event={"ID":"584911f8-e71c-4c68-a868-95949a7bfaa9","Type":"ContainerStarted","Data":"8e9dcaf60df311371b40fcd89bfc3aa06233784079d3bb1b5d1f2b039c9139c0"}
Sep 08 20:26:12.503289 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:12.503228    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br99v" podStartSLOduration=4.91295819 podStartE2EDuration="25.503214492s" totalImagesPullingTime="20.590256302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 20:25:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:26:12.501833181 +0000 UTC m=+25.753314537" watchObservedRunningTime="2026-09-08 20:26:12.503214492 +0000 UTC m=+25.754695846"
Sep 08 20:26:12.503463 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:12.503445    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8fmh6" podStartSLOduration=7.34726207 podStartE2EDuration="25.503438385s" totalImagesPullingTime="18.156176315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 20:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:26:10.48860936 +0000 UTC m=+23.740090716" watchObservedRunningTime="2026-09-08 20:26:12.503438385 +0000 UTC m=+25.754919744"
Sep 08 20:26:13.219653 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:13.219623    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:26:13.219813 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:13.219623    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:26:13.219813 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:13.219733    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwvng" podUID="0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059"
Sep 08 20:26:13.219813 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:13.219799    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5v7c" podUID="0dc52e01-2095-40dd-8603-0fdf9d6ff1aa"
Sep 08 20:26:13.493533 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:13.493302    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb" event={"ID":"4f3b5c49-0e9f-4179-bd36-b019c23a08b7","Type":"ContainerStarted","Data":"d59c31dba666ff4675f2373ba69fb99bdc5c0496fc8ea88195546aa47c0eff36"}
Sep 08 20:26:13.521100 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:13.521053    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb" podStartSLOduration=8.252707302 podStartE2EDuration="26.521037112s" totalImagesPullingTime="18.26832981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 20:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:26:13.520269339 +0000 UTC m=+26.771750695" watchObservedRunningTime="2026-09-08 20:26:13.521037112 +0000 UTC m=+26.772518463"
Sep 08 20:26:14.068311 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:14.068279    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xfczt"
Sep 08 20:26:14.069032 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:14.069009    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xfczt"
Sep 08 20:26:14.219145 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:14.219111    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:26:14.219320 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:14.219262    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8mj2" podUID="cb9818b0-76cd-4be1-9c9e-7d47e0cb1607"
Sep 08 20:26:14.496612 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:14.496519    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:26:14.497045 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:14.496640    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:26:14.497045 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:14.496691    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:26:14.497045 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:14.496892    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xfczt"
Sep 08 20:26:14.512775 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:14.512741    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:26:14.513318 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:14.513299    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:26:15.218804 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:15.218766    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:26:15.218804 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:15.218798    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:26:15.218980 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:15.218940    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwvng" podUID="0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059"
Sep 08 20:26:15.219070 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:15.219041    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5v7c" podUID="0dc52e01-2095-40dd-8603-0fdf9d6ff1aa"
Sep 08 20:26:15.382935 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:15.382824    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g8mj2"]
Sep 08 20:26:15.383102 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:15.382958    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:26:15.383102 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:15.383091    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8mj2" podUID="cb9818b0-76cd-4be1-9c9e-7d47e0cb1607"
Sep 08 20:26:15.383525 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:15.383504    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f5v7c"]
Sep 08 20:26:15.385806 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:15.385781    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zwvng"]
Sep 08 20:26:15.497665 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:15.497643    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:26:15.498375 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:15.497814    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwvng" podUID="0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059"
Sep 08 20:26:15.498375 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:15.497842    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:26:15.498375 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:15.498001    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5v7c" podUID="0dc52e01-2095-40dd-8603-0fdf9d6ff1aa"
Sep 08 20:26:16.501305 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:16.501274    2552 generic.go:356] "Generic (PLEG): container finished" podID="135336c8-5355-4a06-bd24-30cf54fd420a" containerID="02fde3a3adf2772b1b76a9cc5b58f4586de0b200ea7e5fb4948aa5d35a5c09f3" exitCode=0
Sep 08 20:26:16.501836 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:16.501357    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvlkf" event={"ID":"135336c8-5355-4a06-bd24-30cf54fd420a","Type":"ContainerDied","Data":"02fde3a3adf2772b1b76a9cc5b58f4586de0b200ea7e5fb4948aa5d35a5c09f3"}
Sep 08 20:26:17.219699 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:17.219488    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:26:17.219860 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:17.219579    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:26:17.219916 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:17.219887    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwvng" podUID="0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059"
Sep 08 20:26:17.219916 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:17.219770    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8mj2" podUID="cb9818b0-76cd-4be1-9c9e-7d47e0cb1607"
Sep 08 20:26:17.219916 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:17.219630    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:26:17.220115 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:17.220005    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5v7c" podUID="0dc52e01-2095-40dd-8603-0fdf9d6ff1aa"
Sep 08 20:26:18.507260 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:18.507171    2552 generic.go:356] "Generic (PLEG): container finished" podID="135336c8-5355-4a06-bd24-30cf54fd420a" containerID="e28bea2c829b783013b946e8959473f2d47b5bcec467c6ce20571d34121b840b" exitCode=0
Sep 08 20:26:18.507260 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:18.507211    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvlkf" event={"ID":"135336c8-5355-4a06-bd24-30cf54fd420a","Type":"ContainerDied","Data":"e28bea2c829b783013b946e8959473f2d47b5bcec467c6ce20571d34121b840b"}
Sep 08 20:26:19.219021 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:19.218935    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:26:19.219021 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:19.218978    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:26:19.219262 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:19.218943    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:26:19.219262 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:19.219072    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8mj2" podUID="cb9818b0-76cd-4be1-9c9e-7d47e0cb1607"
Sep 08 20:26:19.219262 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:19.219151    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwvng" podUID="0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059"
Sep 08 20:26:19.219262 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:19.219216    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5v7c" podUID="0dc52e01-2095-40dd-8603-0fdf9d6ff1aa"
Sep 08 20:26:20.514491 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:20.514400    2552 generic.go:356] "Generic (PLEG): container finished" podID="135336c8-5355-4a06-bd24-30cf54fd420a" containerID="005e906fe42fb46e1904e0e02ed979ce2686ef9f32fc9159dfe67d063668843c" exitCode=0
Sep 08 20:26:20.514491 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:20.514478    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvlkf" event={"ID":"135336c8-5355-4a06-bd24-30cf54fd420a","Type":"ContainerDied","Data":"005e906fe42fb46e1904e0e02ed979ce2686ef9f32fc9159dfe67d063668843c"}
Sep 08 20:26:20.847989 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:20.847957    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs\") pod \"network-metrics-daemon-zwvng\" (UID: \"0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059\") " pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:26:20.848150 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:20.848082    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 20:26:20.848150 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:20.848143    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs podName:0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059 nodeName:}" failed. No retries permitted until 2026-09-08 20:26:52.848126308 +0000 UTC m=+66.099607646 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs") pod "network-metrics-daemon-zwvng" (UID: "0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 20:26:20.949027 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:20.948994    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ttj\" (UniqueName: \"kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj\") pod \"network-check-target-g8mj2\" (UID: \"cb9818b0-76cd-4be1-9c9e-7d47e0cb1607\") " pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:26:20.949027 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:20.949032    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret\") pod \"global-pull-secret-syncer-f5v7c\" (UID: \"0dc52e01-2095-40dd-8603-0fdf9d6ff1aa\") " pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:26:20.949218 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:20.949144    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 20:26:20.949218 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:20.949159    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 20:26:20.949218 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:20.949174    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 20:26:20.949218 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:20.949187    2552 projected.go:196] Error preparing data for projected volume kube-api-access-x6ttj for pod openshift-network-diagnostics/network-check-target-g8mj2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 20:26:20.949218 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:20.949212    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret podName:0dc52e01-2095-40dd-8603-0fdf9d6ff1aa nodeName:}" failed. No retries permitted until 2026-09-08 20:26:52.949193102 +0000 UTC m=+66.200674457 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret") pod "global-pull-secret-syncer-f5v7c" (UID: "0dc52e01-2095-40dd-8603-0fdf9d6ff1aa") : object "kube-system"/"original-pull-secret" not registered
Sep 08 20:26:20.949402 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:20.949229    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj podName:cb9818b0-76cd-4be1-9c9e-7d47e0cb1607 nodeName:}" failed. No retries permitted until 2026-09-08 20:26:52.949220259 +0000 UTC m=+66.200701595 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x6ttj" (UniqueName: "kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj") pod "network-check-target-g8mj2" (UID: "cb9818b0-76cd-4be1-9c9e-7d47e0cb1607") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 20:26:21.219436 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:21.219368    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:26:21.219594 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:21.219368    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:26:21.219594 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:21.219484    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwvng" podUID="0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059"
Sep 08 20:26:21.219594 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:21.219368    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:26:21.219594 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:21.219550    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5v7c" podUID="0dc52e01-2095-40dd-8603-0fdf9d6ff1aa"
Sep 08 20:26:21.219725 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:21.219620    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8mj2" podUID="cb9818b0-76cd-4be1-9c9e-7d47e0cb1607"
Sep 08 20:26:22.483217 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.483138    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-118.ec2.internal" event="NodeReady"
Sep 08 20:26:22.483633 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.483297    2552 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 20:26:22.530193 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.530161    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pfxtz"]
Sep 08 20:26:22.550514 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.550479    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vdrmg"]
Sep 08 20:26:22.550808 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.550782    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pfxtz"
Sep 08 20:26:22.553342 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.553318    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 20:26:22.553342 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.553369    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 20:26:22.553584 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.553318    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vbwpr\""
Sep 08 20:26:22.568790 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.568745    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pfxtz"]
Sep 08 20:26:22.568790 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.568790    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vdrmg"]
Sep 08 20:26:22.568953 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.568851    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vdrmg"
Sep 08 20:26:22.572562 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.572148    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 20:26:22.572562 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.572309    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 20:26:22.573597 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.572942    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-t62jz\""
Sep 08 20:26:22.573597 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.572994    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 20:26:22.662428 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.662375    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/044d382c-a15c-462b-8cd1-6c7396d46f3a-metrics-tls\") pod \"dns-default-pfxtz\" (UID: \"044d382c-a15c-462b-8cd1-6c7396d46f3a\") " pod="openshift-dns/dns-default-pfxtz"
Sep 08 20:26:22.662617 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.662459    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2c46\" (UniqueName: \"kubernetes.io/projected/044d382c-a15c-462b-8cd1-6c7396d46f3a-kube-api-access-j2c46\") pod \"dns-default-pfxtz\" (UID: \"044d382c-a15c-462b-8cd1-6c7396d46f3a\") " pod="openshift-dns/dns-default-pfxtz"
Sep 08 20:26:22.662617 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.662491    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/044d382c-a15c-462b-8cd1-6c7396d46f3a-tmp-dir\") pod \"dns-default-pfxtz\" (UID: \"044d382c-a15c-462b-8cd1-6c7396d46f3a\") " pod="openshift-dns/dns-default-pfxtz"
Sep 08 20:26:22.662617 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.662541    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/044d382c-a15c-462b-8cd1-6c7396d46f3a-config-volume\") pod \"dns-default-pfxtz\" (UID: \"044d382c-a15c-462b-8cd1-6c7396d46f3a\") " pod="openshift-dns/dns-default-pfxtz"
Sep 08 20:26:22.662617 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.662584    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8c650d9-07af-47a5-80d7-dd191f995c7f-cert\") pod \"ingress-canary-vdrmg\" (UID: \"d8c650d9-07af-47a5-80d7-dd191f995c7f\") " pod="openshift-ingress-canary/ingress-canary-vdrmg"
Sep 08 20:26:22.662840 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.662622    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj5k2\" (UniqueName: \"kubernetes.io/projected/d8c650d9-07af-47a5-80d7-dd191f995c7f-kube-api-access-nj5k2\") pod \"ingress-canary-vdrmg\" (UID: \"d8c650d9-07af-47a5-80d7-dd191f995c7f\") " pod="openshift-ingress-canary/ingress-canary-vdrmg"
Sep 08 20:26:22.763761 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.763680    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2c46\" (UniqueName: \"kubernetes.io/projected/044d382c-a15c-462b-8cd1-6c7396d46f3a-kube-api-access-j2c46\") pod \"dns-default-pfxtz\" (UID: \"044d382c-a15c-462b-8cd1-6c7396d46f3a\") " pod="openshift-dns/dns-default-pfxtz"
Sep 08 20:26:22.763761 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.763720    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/044d382c-a15c-462b-8cd1-6c7396d46f3a-tmp-dir\") pod \"dns-default-pfxtz\" (UID: \"044d382c-a15c-462b-8cd1-6c7396d46f3a\") " pod="openshift-dns/dns-default-pfxtz"
Sep 08 20:26:22.763761 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.763756    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/044d382c-a15c-462b-8cd1-6c7396d46f3a-config-volume\") pod \"dns-default-pfxtz\" (UID: \"044d382c-a15c-462b-8cd1-6c7396d46f3a\") " pod="openshift-dns/dns-default-pfxtz"
Sep 08 20:26:22.763976 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.763772    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8c650d9-07af-47a5-80d7-dd191f995c7f-cert\") pod \"ingress-canary-vdrmg\" (UID: \"d8c650d9-07af-47a5-80d7-dd191f995c7f\") " pod="openshift-ingress-canary/ingress-canary-vdrmg"
Sep 08 20:26:22.763976 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.763788    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj5k2\" (UniqueName: \"kubernetes.io/projected/d8c650d9-07af-47a5-80d7-dd191f995c7f-kube-api-access-nj5k2\") pod \"ingress-canary-vdrmg\" (UID: \"d8c650d9-07af-47a5-80d7-dd191f995c7f\") " pod="openshift-ingress-canary/ingress-canary-vdrmg"
Sep 08 20:26:22.763976 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.763804    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/044d382c-a15c-462b-8cd1-6c7396d46f3a-metrics-tls\") pod \"dns-default-pfxtz\" (UID: \"044d382c-a15c-462b-8cd1-6c7396d46f3a\") " pod="openshift-dns/dns-default-pfxtz"
Sep 08 20:26:22.764160 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.764141    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/044d382c-a15c-462b-8cd1-6c7396d46f3a-tmp-dir\") pod \"dns-default-pfxtz\" (UID: \"044d382c-a15c-462b-8cd1-6c7396d46f3a\") " pod="openshift-dns/dns-default-pfxtz"
Sep 08 20:26:22.764199 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:22.764148    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 20:26:22.764259 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:22.764224    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/044d382c-a15c-462b-8cd1-6c7396d46f3a-metrics-tls podName:044d382c-a15c-462b-8cd1-6c7396d46f3a nodeName:}" failed. No retries permitted until 2026-09-08 20:26:23.264207642 +0000 UTC m=+36.515688976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/044d382c-a15c-462b-8cd1-6c7396d46f3a-metrics-tls") pod "dns-default-pfxtz" (UID: "044d382c-a15c-462b-8cd1-6c7396d46f3a") : secret "dns-default-metrics-tls" not found
Sep 08 20:26:22.764774 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.764745    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/044d382c-a15c-462b-8cd1-6c7396d46f3a-config-volume\") pod \"dns-default-pfxtz\" (UID: \"044d382c-a15c-462b-8cd1-6c7396d46f3a\") " pod="openshift-dns/dns-default-pfxtz"
Sep 08 20:26:22.768166 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.768049    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8c650d9-07af-47a5-80d7-dd191f995c7f-cert\") pod \"ingress-canary-vdrmg\" (UID: \"d8c650d9-07af-47a5-80d7-dd191f995c7f\") " pod="openshift-ingress-canary/ingress-canary-vdrmg"
Sep 08 20:26:22.772738 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.772710    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj5k2\" (UniqueName: \"kubernetes.io/projected/d8c650d9-07af-47a5-80d7-dd191f995c7f-kube-api-access-nj5k2\") pod \"ingress-canary-vdrmg\" (UID: \"d8c650d9-07af-47a5-80d7-dd191f995c7f\") " pod="openshift-ingress-canary/ingress-canary-vdrmg"
Sep 08 20:26:22.772966 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.772942    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2c46\" (UniqueName: \"kubernetes.io/projected/044d382c-a15c-462b-8cd1-6c7396d46f3a-kube-api-access-j2c46\") pod \"dns-default-pfxtz\" (UID: \"044d382c-a15c-462b-8cd1-6c7396d46f3a\") " pod="openshift-dns/dns-default-pfxtz"
Sep 08 20:26:22.882287 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:22.882232    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vdrmg"
Sep 08 20:26:23.219045 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:23.219005    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:26:23.219228 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:23.219005    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:26:23.219228 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:23.219014    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:26:23.222020 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:23.221987    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 20:26:23.224055 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:23.224032    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2nzg7\""
Sep 08 20:26:23.224162 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:23.224065    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kbhtv\""
Sep 08 20:26:23.224162 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:23.224070    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 20:26:23.224384 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:23.224366    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 20:26:23.224434 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:23.224396    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 20:26:23.224524 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:23.224491    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vdrmg"]
Sep 08 20:26:23.226586 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:26:23.226564    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c650d9_07af_47a5_80d7_dd191f995c7f.slice/crio-2771dc4c2dc9b51ce0145046dcbeec7d2a57504bd7d33e68c4e7a013fcf3a1a3 WatchSource:0}: Error finding container 2771dc4c2dc9b51ce0145046dcbeec7d2a57504bd7d33e68c4e7a013fcf3a1a3: Status 404 returned error can't find the container with id 2771dc4c2dc9b51ce0145046dcbeec7d2a57504bd7d33e68c4e7a013fcf3a1a3
Sep 08 20:26:23.267455 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:23.267420    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/044d382c-a15c-462b-8cd1-6c7396d46f3a-metrics-tls\") pod \"dns-default-pfxtz\" (UID: \"044d382c-a15c-462b-8cd1-6c7396d46f3a\") " pod="openshift-dns/dns-default-pfxtz"
Sep 08 20:26:23.267594 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:23.267578    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 20:26:23.267665 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:23.267654    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/044d382c-a15c-462b-8cd1-6c7396d46f3a-metrics-tls podName:044d382c-a15c-462b-8cd1-6c7396d46f3a nodeName:}" failed. No retries permitted until 2026-09-08 20:26:24.267636182 +0000 UTC m=+37.519117520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/044d382c-a15c-462b-8cd1-6c7396d46f3a-metrics-tls") pod "dns-default-pfxtz" (UID: "044d382c-a15c-462b-8cd1-6c7396d46f3a") : secret "dns-default-metrics-tls" not found
Sep 08 20:26:23.522737 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:23.522657    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vdrmg" event={"ID":"d8c650d9-07af-47a5-80d7-dd191f995c7f","Type":"ContainerStarted","Data":"2771dc4c2dc9b51ce0145046dcbeec7d2a57504bd7d33e68c4e7a013fcf3a1a3"}
Sep 08 20:26:24.275414 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:24.275384    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/044d382c-a15c-462b-8cd1-6c7396d46f3a-metrics-tls\") pod \"dns-default-pfxtz\" (UID: \"044d382c-a15c-462b-8cd1-6c7396d46f3a\") " pod="openshift-dns/dns-default-pfxtz"
Sep 08 20:26:24.275595 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:24.275573    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 20:26:24.275663 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:24.275656    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/044d382c-a15c-462b-8cd1-6c7396d46f3a-metrics-tls podName:044d382c-a15c-462b-8cd1-6c7396d46f3a nodeName:}" failed. No retries permitted until 2026-09-08 20:26:26.275634284 +0000 UTC m=+39.527115633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/044d382c-a15c-462b-8cd1-6c7396d46f3a-metrics-tls") pod "dns-default-pfxtz" (UID: "044d382c-a15c-462b-8cd1-6c7396d46f3a") : secret "dns-default-metrics-tls" not found
Sep 08 20:26:25.486749 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:25.486678    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kw8zt"]
Sep 08 20:26:25.500022 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:25.499990    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:25.501734 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:25.501708    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kw8zt"]
Sep 08 20:26:25.502628 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:25.502567    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 20:26:25.502628 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:25.502624    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 20:26:25.502830 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:25.502671    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 20:26:25.502970 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:25.502953    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 20:26:25.503059 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:25.502952    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4gd8m\""
Sep 08 20:26:25.535507 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:25.535474    2552 generic.go:356] "Generic (PLEG): container finished" podID="135336c8-5355-4a06-bd24-30cf54fd420a" containerID="450dc1a24083299a7a25b578db31d6a04f3c2d1703ebf70472385258a94d63f2" exitCode=0
Sep 08 20:26:25.535649 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:25.535545    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvlkf" event={"ID":"135336c8-5355-4a06-bd24-30cf54fd420a","Type":"ContainerDied","Data":"450dc1a24083299a7a25b578db31d6a04f3c2d1703ebf70472385258a94d63f2"}
Sep 08 20:26:25.585616 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:25.585581    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59pkc\" (UniqueName: \"kubernetes.io/projected/607dd30a-5e39-4cc8-91b0-cba0ad740873-kube-api-access-59pkc\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:25.585763 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:25.585640    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/607dd30a-5e39-4cc8-91b0-cba0ad740873-crio-socket\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:25.585763 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:25.585734    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/607dd30a-5e39-4cc8-91b0-cba0ad740873-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:25.585866 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:25.585774    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/607dd30a-5e39-4cc8-91b0-cba0ad740873-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:25.585866 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:25.585806    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/607dd30a-5e39-4cc8-91b0-cba0ad740873-data-volume\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:25.686759 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:25.686734    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59pkc\" (UniqueName: \"kubernetes.io/projected/607dd30a-5e39-4cc8-91b0-cba0ad740873-kube-api-access-59pkc\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:25.686883 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:25.686784    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/607dd30a-5e39-4cc8-91b0-cba0ad740873-crio-socket\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:25.686883 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:25.686839    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/607dd30a-5e39-4cc8-91b0-cba0ad740873-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:25.686883 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:25.686873    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/607dd30a-5e39-4cc8-91b0-cba0ad740873-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:25.687039 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:25.686899    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/607dd30a-5e39-4cc8-91b0-cba0ad740873-data-volume\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:25.687161 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:25.687129    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 20:26:25.687260 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:25.687198    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/607dd30a-5e39-4cc8-91b0-cba0ad740873-kube-rbac-proxy-cm podName:607dd30a-5e39-4cc8-91b0-cba0ad740873 nodeName:}" failed. No retries permitted until 2026-09-08 20:26:26.187178152 +0000 UTC m=+39.438659505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/607dd30a-5e39-4cc8-91b0-cba0ad740873-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kw8zt" (UID: "607dd30a-5e39-4cc8-91b0-cba0ad740873") : configmap "kube-rbac-proxy" not found
Sep 08 20:26:25.687324 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:25.687310    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/607dd30a-5e39-4cc8-91b0-cba0ad740873-crio-socket\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:25.687384 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:25.687314    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 20:26:25.687384 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:25.687379    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/607dd30a-5e39-4cc8-91b0-cba0ad740873-insights-runtime-extractor-tls podName:607dd30a-5e39-4cc8-91b0-cba0ad740873 nodeName:}" failed. No retries permitted until 2026-09-08 20:26:26.187362572 +0000 UTC m=+39.438843907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/607dd30a-5e39-4cc8-91b0-cba0ad740873-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kw8zt" (UID: "607dd30a-5e39-4cc8-91b0-cba0ad740873") : secret "insights-runtime-extractor-tls" not found
Sep 08 20:26:25.687477 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:25.687394    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/607dd30a-5e39-4cc8-91b0-cba0ad740873-data-volume\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:25.695630 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:25.695476    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59pkc\" (UniqueName: \"kubernetes.io/projected/607dd30a-5e39-4cc8-91b0-cba0ad740873-kube-api-access-59pkc\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:26.191920 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:26.191870    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/607dd30a-5e39-4cc8-91b0-cba0ad740873-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:26.191920 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:26.191930    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/607dd30a-5e39-4cc8-91b0-cba0ad740873-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:26.192166 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:26.192013    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 20:26:26.192166 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:26.192084    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 20:26:26.192166 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:26.192091    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/607dd30a-5e39-4cc8-91b0-cba0ad740873-kube-rbac-proxy-cm podName:607dd30a-5e39-4cc8-91b0-cba0ad740873 nodeName:}" failed. No retries permitted until 2026-09-08 20:26:27.192074135 +0000 UTC m=+40.443555493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/607dd30a-5e39-4cc8-91b0-cba0ad740873-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kw8zt" (UID: "607dd30a-5e39-4cc8-91b0-cba0ad740873") : configmap "kube-rbac-proxy" not found
Sep 08 20:26:26.192166 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:26.192136    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/607dd30a-5e39-4cc8-91b0-cba0ad740873-insights-runtime-extractor-tls podName:607dd30a-5e39-4cc8-91b0-cba0ad740873 nodeName:}" failed. No retries permitted until 2026-09-08 20:26:27.192124165 +0000 UTC m=+40.443605499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/607dd30a-5e39-4cc8-91b0-cba0ad740873-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kw8zt" (UID: "607dd30a-5e39-4cc8-91b0-cba0ad740873") : secret "insights-runtime-extractor-tls" not found
Sep 08 20:26:26.292721 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:26.292687    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/044d382c-a15c-462b-8cd1-6c7396d46f3a-metrics-tls\") pod \"dns-default-pfxtz\" (UID: \"044d382c-a15c-462b-8cd1-6c7396d46f3a\") " pod="openshift-dns/dns-default-pfxtz"
Sep 08 20:26:26.292884 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:26.292847    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 20:26:26.292924 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:26.292908    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/044d382c-a15c-462b-8cd1-6c7396d46f3a-metrics-tls podName:044d382c-a15c-462b-8cd1-6c7396d46f3a nodeName:}" failed. No retries permitted until 2026-09-08 20:26:30.292892971 +0000 UTC m=+43.544374332 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/044d382c-a15c-462b-8cd1-6c7396d46f3a-metrics-tls") pod "dns-default-pfxtz" (UID: "044d382c-a15c-462b-8cd1-6c7396d46f3a") : secret "dns-default-metrics-tls" not found
Sep 08 20:26:26.541428 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:26.541347    2552 generic.go:356] "Generic (PLEG): container finished" podID="135336c8-5355-4a06-bd24-30cf54fd420a" containerID="8a15bbfd454bbbf185a1a052add5354f1fb35a74908a01f5162f1b5d47184d68" exitCode=0
Sep 08 20:26:26.541428 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:26.541401    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvlkf" event={"ID":"135336c8-5355-4a06-bd24-30cf54fd420a","Type":"ContainerDied","Data":"8a15bbfd454bbbf185a1a052add5354f1fb35a74908a01f5162f1b5d47184d68"}
Sep 08 20:26:26.632062 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:26.632036    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-n78ql"]
Sep 08 20:26:26.652025 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:26.651994    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-n78ql"]
Sep 08 20:26:26.652159 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:26.652136    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-n78ql"
Sep 08 20:26:26.654693 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:26.654673    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 08 20:26:26.654693 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:26.654675    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 08 20:26:26.654883 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:26.654680    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 08 20:26:26.654883 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:26.654781    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-98lf8\""
Sep 08 20:26:26.654985 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:26.654961    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 08 20:26:26.797367 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:26.797262    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cc635cce-41dc-4043-8f41-b7f395037fb0-signing-cabundle\") pod \"service-ca-75d99c484-n78ql\" (UID: \"cc635cce-41dc-4043-8f41-b7f395037fb0\") " pod="openshift-service-ca/service-ca-75d99c484-n78ql"
Sep 08 20:26:26.797367 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:26.797299    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkdr9\" (UniqueName: \"kubernetes.io/projected/cc635cce-41dc-4043-8f41-b7f395037fb0-kube-api-access-wkdr9\") pod \"service-ca-75d99c484-n78ql\" (UID: \"cc635cce-41dc-4043-8f41-b7f395037fb0\") " pod="openshift-service-ca/service-ca-75d99c484-n78ql"
Sep 08 20:26:26.797367 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:26.797324    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cc635cce-41dc-4043-8f41-b7f395037fb0-signing-key\") pod \"service-ca-75d99c484-n78ql\" (UID: \"cc635cce-41dc-4043-8f41-b7f395037fb0\") " pod="openshift-service-ca/service-ca-75d99c484-n78ql"
Sep 08 20:26:26.898580 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:26.898534    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cc635cce-41dc-4043-8f41-b7f395037fb0-signing-cabundle\") pod \"service-ca-75d99c484-n78ql\" (UID: \"cc635cce-41dc-4043-8f41-b7f395037fb0\") " pod="openshift-service-ca/service-ca-75d99c484-n78ql"
Sep 08 20:26:26.898580 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:26.898588    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdr9\" (UniqueName: \"kubernetes.io/projected/cc635cce-41dc-4043-8f41-b7f395037fb0-kube-api-access-wkdr9\") pod \"service-ca-75d99c484-n78ql\" (UID: \"cc635cce-41dc-4043-8f41-b7f395037fb0\") " pod="openshift-service-ca/service-ca-75d99c484-n78ql"
Sep 08 20:26:26.898816 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:26.898623    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cc635cce-41dc-4043-8f41-b7f395037fb0-signing-key\") pod \"service-ca-75d99c484-n78ql\" (UID: \"cc635cce-41dc-4043-8f41-b7f395037fb0\") " pod="openshift-service-ca/service-ca-75d99c484-n78ql"
Sep 08 20:26:26.899402 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:26.899376    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cc635cce-41dc-4043-8f41-b7f395037fb0-signing-cabundle\") pod \"service-ca-75d99c484-n78ql\" (UID: \"cc635cce-41dc-4043-8f41-b7f395037fb0\") " pod="openshift-service-ca/service-ca-75d99c484-n78ql"
Sep 08 20:26:26.902089 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:26.902064    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cc635cce-41dc-4043-8f41-b7f395037fb0-signing-key\") pod \"service-ca-75d99c484-n78ql\" (UID: \"cc635cce-41dc-4043-8f41-b7f395037fb0\") " pod="openshift-service-ca/service-ca-75d99c484-n78ql"
Sep 08 20:26:26.907484 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:26.907462    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkdr9\" (UniqueName: \"kubernetes.io/projected/cc635cce-41dc-4043-8f41-b7f395037fb0-kube-api-access-wkdr9\") pod \"service-ca-75d99c484-n78ql\" (UID: \"cc635cce-41dc-4043-8f41-b7f395037fb0\") " pod="openshift-service-ca/service-ca-75d99c484-n78ql"
Sep 08 20:26:26.961362 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:26.961329    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-n78ql"
Sep 08 20:26:27.201254 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:27.201203    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/607dd30a-5e39-4cc8-91b0-cba0ad740873-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:27.201437 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:27.201270    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/607dd30a-5e39-4cc8-91b0-cba0ad740873-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:27.201437 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:27.201332    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 20:26:27.201437 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:27.201393    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 20:26:27.201437 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:27.201412    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/607dd30a-5e39-4cc8-91b0-cba0ad740873-kube-rbac-proxy-cm podName:607dd30a-5e39-4cc8-91b0-cba0ad740873 nodeName:}" failed. No retries permitted until 2026-09-08 20:26:29.201388679 +0000 UTC m=+42.452870022 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/607dd30a-5e39-4cc8-91b0-cba0ad740873-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kw8zt" (UID: "607dd30a-5e39-4cc8-91b0-cba0ad740873") : configmap "kube-rbac-proxy" not found
Sep 08 20:26:27.201437 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:27.201440    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/607dd30a-5e39-4cc8-91b0-cba0ad740873-insights-runtime-extractor-tls podName:607dd30a-5e39-4cc8-91b0-cba0ad740873 nodeName:}" failed. No retries permitted until 2026-09-08 20:26:29.201428722 +0000 UTC m=+42.452910057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/607dd30a-5e39-4cc8-91b0-cba0ad740873-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kw8zt" (UID: "607dd30a-5e39-4cc8-91b0-cba0ad740873") : secret "insights-runtime-extractor-tls" not found
Sep 08 20:26:27.547269 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:27.547219    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvlkf" event={"ID":"135336c8-5355-4a06-bd24-30cf54fd420a","Type":"ContainerStarted","Data":"8ac5919c55cfb46b1fcc113358915a6bfaf724eaa6ae08020189c8ae8f14961b"}
Sep 08 20:26:27.572222 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:27.572176    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zvlkf" podStartSLOduration=9.438225489 podStartE2EDuration="40.572160713s" totalImagesPullingTime="31.133935224s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 20:25:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:26:27.57076216 +0000 UTC m=+40.822243541" watchObservedRunningTime="2026-09-08 20:26:27.572160713 +0000 UTC m=+40.823642241"
Sep 08 20:26:27.633855 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:27.633833    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-th7xh_bced89cc-7b97-493f-b71d-8fc32bc7187f/dns-node-resolver/0.log"
Sep 08 20:26:27.706816 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:27.706394    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-n78ql"]
Sep 08 20:26:27.739943 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:26:27.739883    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc635cce_41dc_4043_8f41_b7f395037fb0.slice/crio-21b31352b913d6abba7eb5494a6fae6d72c28fb850e8c1e82fcd74159827d83e WatchSource:0}: Error finding container 21b31352b913d6abba7eb5494a6fae6d72c28fb850e8c1e82fcd74159827d83e: Status 404 returned error can't find the container with id 21b31352b913d6abba7eb5494a6fae6d72c28fb850e8c1e82fcd74159827d83e
Sep 08 20:26:28.551151 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:28.551110    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-n78ql" event={"ID":"cc635cce-41dc-4043-8f41-b7f395037fb0","Type":"ContainerStarted","Data":"21b31352b913d6abba7eb5494a6fae6d72c28fb850e8c1e82fcd74159827d83e"}
Sep 08 20:26:28.552954 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:28.552926    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vdrmg" event={"ID":"d8c650d9-07af-47a5-80d7-dd191f995c7f","Type":"ContainerStarted","Data":"e86e1703c6e87ed133c8734904176bd565ca25efed8f7ee10289b2118e4abcf2"}
Sep 08 20:26:28.570130 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:28.570078    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vdrmg" podStartSLOduration=2.028500417 podStartE2EDuration="6.570064616s" totalImagesPullingTime="4.541564199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:26:28.569334614 +0000 UTC m=+41.820815984" watchObservedRunningTime="2026-09-08 20:26:28.570064616 +0000 UTC m=+41.821545982"
Sep 08 20:26:28.633707 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:28.633675    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zxmp4_345de38f-94c6-4067-a5c6-fc7ad6a560e9/node-ca/0.log"
Sep 08 20:26:29.217070 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:29.217024    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/607dd30a-5e39-4cc8-91b0-cba0ad740873-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:29.217262 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:29.217091    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/607dd30a-5e39-4cc8-91b0-cba0ad740873-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:29.217262 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:29.217157    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 20:26:29.217262 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:29.217224    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 20:26:29.217262 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:29.217253    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/607dd30a-5e39-4cc8-91b0-cba0ad740873-kube-rbac-proxy-cm podName:607dd30a-5e39-4cc8-91b0-cba0ad740873 nodeName:}" failed. No retries permitted until 2026-09-08 20:26:33.217212011 +0000 UTC m=+46.468693375 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/607dd30a-5e39-4cc8-91b0-cba0ad740873-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kw8zt" (UID: "607dd30a-5e39-4cc8-91b0-cba0ad740873") : configmap "kube-rbac-proxy" not found
Sep 08 20:26:29.217490 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:29.217295    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/607dd30a-5e39-4cc8-91b0-cba0ad740873-insights-runtime-extractor-tls podName:607dd30a-5e39-4cc8-91b0-cba0ad740873 nodeName:}" failed. No retries permitted until 2026-09-08 20:26:33.217278542 +0000 UTC m=+46.468759879 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/607dd30a-5e39-4cc8-91b0-cba0ad740873-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kw8zt" (UID: "607dd30a-5e39-4cc8-91b0-cba0ad740873") : secret "insights-runtime-extractor-tls" not found
Sep 08 20:26:29.433383 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:29.433354    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vdrmg_d8c650d9-07af-47a5-80d7-dd191f995c7f/serve-healthcheck-canary/0.log"
Sep 08 20:26:30.324420 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:30.324332    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/044d382c-a15c-462b-8cd1-6c7396d46f3a-metrics-tls\") pod \"dns-default-pfxtz\" (UID: \"044d382c-a15c-462b-8cd1-6c7396d46f3a\") " pod="openshift-dns/dns-default-pfxtz"
Sep 08 20:26:30.324761 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:30.324482    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 20:26:30.324761 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:30.324548    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/044d382c-a15c-462b-8cd1-6c7396d46f3a-metrics-tls podName:044d382c-a15c-462b-8cd1-6c7396d46f3a nodeName:}" failed. No retries permitted until 2026-09-08 20:26:38.324531506 +0000 UTC m=+51.576012844 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/044d382c-a15c-462b-8cd1-6c7396d46f3a-metrics-tls") pod "dns-default-pfxtz" (UID: "044d382c-a15c-462b-8cd1-6c7396d46f3a") : secret "dns-default-metrics-tls" not found
Sep 08 20:26:30.559667 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:30.559634    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-n78ql" event={"ID":"cc635cce-41dc-4043-8f41-b7f395037fb0","Type":"ContainerStarted","Data":"b216fa1032a04f8a692e6fd93855e242064b516a9fb954dec3bb9a3eedfcb662"}
Sep 08 20:26:33.247285 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:33.247230    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/607dd30a-5e39-4cc8-91b0-cba0ad740873-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:33.247729 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:33.247305    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/607dd30a-5e39-4cc8-91b0-cba0ad740873-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:33.247729 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:33.247319    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 20:26:33.247729 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:33.247382    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/607dd30a-5e39-4cc8-91b0-cba0ad740873-kube-rbac-proxy-cm podName:607dd30a-5e39-4cc8-91b0-cba0ad740873 nodeName:}" failed. No retries permitted until 2026-09-08 20:26:41.247361241 +0000 UTC m=+54.498842597 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/607dd30a-5e39-4cc8-91b0-cba0ad740873-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kw8zt" (UID: "607dd30a-5e39-4cc8-91b0-cba0ad740873") : configmap "kube-rbac-proxy" not found
Sep 08 20:26:33.247729 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:33.247455    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 20:26:33.247729 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:33.247514    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/607dd30a-5e39-4cc8-91b0-cba0ad740873-insights-runtime-extractor-tls podName:607dd30a-5e39-4cc8-91b0-cba0ad740873 nodeName:}" failed. No retries permitted until 2026-09-08 20:26:41.247499145 +0000 UTC m=+54.498980483 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/607dd30a-5e39-4cc8-91b0-cba0ad740873-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kw8zt" (UID: "607dd30a-5e39-4cc8-91b0-cba0ad740873") : secret "insights-runtime-extractor-tls" not found
Sep 08 20:26:38.385628 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:38.385596    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/044d382c-a15c-462b-8cd1-6c7396d46f3a-metrics-tls\") pod \"dns-default-pfxtz\" (UID: \"044d382c-a15c-462b-8cd1-6c7396d46f3a\") " pod="openshift-dns/dns-default-pfxtz"
Sep 08 20:26:38.387807 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:38.387785    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/044d382c-a15c-462b-8cd1-6c7396d46f3a-metrics-tls\") pod \"dns-default-pfxtz\" (UID: \"044d382c-a15c-462b-8cd1-6c7396d46f3a\") " pod="openshift-dns/dns-default-pfxtz"
Sep 08 20:26:38.461931 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:38.461907    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pfxtz"
Sep 08 20:26:38.732444 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:38.732338    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-n78ql" podStartSLOduration=10.445004987 podStartE2EDuration="12.732314606s" totalImagesPullingTime="2.287309619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:26:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:26:30.577494363 +0000 UTC m=+43.828975719" watchObservedRunningTime="2026-09-08 20:26:38.732314606 +0000 UTC m=+51.983795962"
Sep 08 20:26:38.732637 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:38.732532    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pfxtz"]
Sep 08 20:26:39.584771 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:39.584715    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pfxtz" event={"ID":"044d382c-a15c-462b-8cd1-6c7396d46f3a","Type":"ContainerStarted","Data":"616d8fd4ce0e1f3dfe33ea95e5471b8364de897ef9d055b751d4225083bdd75b"}
Sep 08 20:26:41.309560 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:41.309511    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/607dd30a-5e39-4cc8-91b0-cba0ad740873-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:41.309560 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:41.309562    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/607dd30a-5e39-4cc8-91b0-cba0ad740873-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:41.309976 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:41.309688    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 20:26:41.309976 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:26:41.309753    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/607dd30a-5e39-4cc8-91b0-cba0ad740873-insights-runtime-extractor-tls podName:607dd30a-5e39-4cc8-91b0-cba0ad740873 nodeName:}" failed. No retries permitted until 2026-09-08 20:26:57.309735026 +0000 UTC m=+70.561216365 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/607dd30a-5e39-4cc8-91b0-cba0ad740873-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kw8zt" (UID: "607dd30a-5e39-4cc8-91b0-cba0ad740873") : secret "insights-runtime-extractor-tls" not found
Sep 08 20:26:41.310155 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:41.310134    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/607dd30a-5e39-4cc8-91b0-cba0ad740873-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:41.592709 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:41.592678    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pfxtz" event={"ID":"044d382c-a15c-462b-8cd1-6c7396d46f3a","Type":"ContainerStarted","Data":"d5d0d659967ec1c766de582254611c648a9685a21aa3b9250b03337bef0fa9cc"}
Sep 08 20:26:41.592859 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:41.592730    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pfxtz" event={"ID":"044d382c-a15c-462b-8cd1-6c7396d46f3a","Type":"ContainerStarted","Data":"4a1998d01065a12ca094150d13a8dd0ee9dee920165ad11480c14fc175ad11d0"}
Sep 08 20:26:41.592859 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:41.592805    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pfxtz"
Sep 08 20:26:41.610696 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:41.610650    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pfxtz" podStartSLOduration=17.781792621 podStartE2EDuration="19.610634868s" totalImagesPullingTime="1.828842247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:26:41.610015043 +0000 UTC m=+54.861496400" watchObservedRunningTime="2026-09-08 20:26:41.610634868 +0000 UTC m=+54.862116218"
Sep 08 20:26:46.512252 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:46.512210    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5kdmb"
Sep 08 20:26:47.385085 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.385053    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-lghl7"]
Sep 08 20:26:47.417937 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.417906    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d998b5f7d-6bl45"]
Sep 08 20:26:47.418105 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.418055    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-lghl7"
Sep 08 20:26:47.421537 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.421515    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 08 20:26:47.421720 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.421700    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-xrphv\""
Sep 08 20:26:47.421809 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.421703    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 08 20:26:47.430565 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.430541    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw"]
Sep 08 20:26:47.430690 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.430673    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d998b5f7d-6bl45"
Sep 08 20:26:47.437074 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.437053    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 08 20:26:47.437661 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.437640    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 08 20:26:47.437804 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.437788    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 08 20:26:47.437872 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.437859    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 08 20:26:47.438003 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.437988    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-tc7pm\""
Sep 08 20:26:47.448866 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.448842    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-lghl7"]
Sep 08 20:26:47.448972 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.448881    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d998b5f7d-6bl45"]
Sep 08 20:26:47.448972 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.448949    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw"
Sep 08 20:26:47.448972 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.448967    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw"]
Sep 08 20:26:47.451649 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.451626    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 08 20:26:47.451887 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.451872    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 08 20:26:47.452300 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.452284    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 08 20:26:47.453523 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.453497    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2393af3f-39b6-481d-bb56-5333012ace73-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d998b5f7d-6bl45\" (UID: \"2393af3f-39b6-481d-bb56-5333012ace73\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d998b5f7d-6bl45"
Sep 08 20:26:47.453630 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.453562    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxh47\" (UniqueName: \"kubernetes.io/projected/2393af3f-39b6-481d-bb56-5333012ace73-kube-api-access-jxh47\") pod \"managed-serviceaccount-addon-agent-7d998b5f7d-6bl45\" (UID: \"2393af3f-39b6-481d-bb56-5333012ace73\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d998b5f7d-6bl45"
Sep 08 20:26:47.453690 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.453657    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs6xt\" (UniqueName: \"kubernetes.io/projected/3fa62b2d-92ab-4958-83a4-5fa1c56ea4cd-kube-api-access-hs6xt\") pod \"migrator-fdbb8fdff-lghl7\" (UID: \"3fa62b2d-92ab-4958-83a4-5fa1c56ea4cd\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-lghl7"
Sep 08 20:26:47.454731 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.454712    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 08 20:26:47.554678 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.554647    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs6xt\" (UniqueName: \"kubernetes.io/projected/3fa62b2d-92ab-4958-83a4-5fa1c56ea4cd-kube-api-access-hs6xt\") pod \"migrator-fdbb8fdff-lghl7\" (UID: \"3fa62b2d-92ab-4958-83a4-5fa1c56ea4cd\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-lghl7"
Sep 08 20:26:47.555074 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.554683    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2393af3f-39b6-481d-bb56-5333012ace73-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d998b5f7d-6bl45\" (UID: \"2393af3f-39b6-481d-bb56-5333012ace73\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d998b5f7d-6bl45"
Sep 08 20:26:47.555074 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.554716    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7bb4687d54-bgcmw\" (UID: \"1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw"
Sep 08 20:26:47.555074 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.554733    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6zvz\" (UniqueName: \"kubernetes.io/projected/1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5-kube-api-access-t6zvz\") pod \"cluster-proxy-proxy-agent-7bb4687d54-bgcmw\" (UID: \"1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw"
Sep 08 20:26:47.555074 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.554762    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxh47\" (UniqueName: \"kubernetes.io/projected/2393af3f-39b6-481d-bb56-5333012ace73-kube-api-access-jxh47\") pod \"managed-serviceaccount-addon-agent-7d998b5f7d-6bl45\" (UID: \"2393af3f-39b6-481d-bb56-5333012ace73\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d998b5f7d-6bl45"
Sep 08 20:26:47.555074 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.554786    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5-ca\") pod \"cluster-proxy-proxy-agent-7bb4687d54-bgcmw\" (UID: \"1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw"
Sep 08 20:26:47.555074 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.554840    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7bb4687d54-bgcmw\" (UID: \"1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw"
Sep 08 20:26:47.555074 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.554878    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5-hub\") pod \"cluster-proxy-proxy-agent-7bb4687d54-bgcmw\" (UID: \"1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw"
Sep 08 20:26:47.555074 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.554905    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7bb4687d54-bgcmw\" (UID: \"1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw"
Sep 08 20:26:47.557137 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.557114    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2393af3f-39b6-481d-bb56-5333012ace73-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d998b5f7d-6bl45\" (UID: \"2393af3f-39b6-481d-bb56-5333012ace73\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d998b5f7d-6bl45"
Sep 08 20:26:47.567080 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.567060    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs6xt\" (UniqueName: \"kubernetes.io/projected/3fa62b2d-92ab-4958-83a4-5fa1c56ea4cd-kube-api-access-hs6xt\") pod \"migrator-fdbb8fdff-lghl7\" (UID: \"3fa62b2d-92ab-4958-83a4-5fa1c56ea4cd\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-lghl7"
Sep 08 20:26:47.567669 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.567646    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxh47\" (UniqueName: \"kubernetes.io/projected/2393af3f-39b6-481d-bb56-5333012ace73-kube-api-access-jxh47\") pod \"managed-serviceaccount-addon-agent-7d998b5f7d-6bl45\" (UID: \"2393af3f-39b6-481d-bb56-5333012ace73\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d998b5f7d-6bl45"
Sep 08 20:26:47.655775 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.655692    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7bb4687d54-bgcmw\" (UID: \"1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw"
Sep 08 20:26:47.655775 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.655731    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6zvz\" (UniqueName: \"kubernetes.io/projected/1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5-kube-api-access-t6zvz\") pod \"cluster-proxy-proxy-agent-7bb4687d54-bgcmw\" (UID: \"1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw"
Sep 08 20:26:47.655982 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.655781    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5-ca\") pod \"cluster-proxy-proxy-agent-7bb4687d54-bgcmw\" (UID: \"1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw"
Sep 08 20:26:47.655982 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.655815    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7bb4687d54-bgcmw\" (UID: \"1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw"
Sep 08 20:26:47.655982 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.655838    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5-hub\") pod \"cluster-proxy-proxy-agent-7bb4687d54-bgcmw\" (UID: \"1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw"
Sep 08 20:26:47.655982 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.655860    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7bb4687d54-bgcmw\" (UID: \"1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw"
Sep 08 20:26:47.656622 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.656588    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7bb4687d54-bgcmw\" (UID: \"1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw"
Sep 08 20:26:47.658452 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.658429    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5-ca\") pod \"cluster-proxy-proxy-agent-7bb4687d54-bgcmw\" (UID: \"1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw"
Sep 08 20:26:47.658548 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.658487    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5-hub\") pod \"cluster-proxy-proxy-agent-7bb4687d54-bgcmw\" (UID: \"1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw"
Sep 08 20:26:47.658548 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.658499    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7bb4687d54-bgcmw\" (UID: \"1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw"
Sep 08 20:26:47.658620 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.658603    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7bb4687d54-bgcmw\" (UID: \"1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw"
Sep 08 20:26:47.671281 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.671230    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6zvz\" (UniqueName: \"kubernetes.io/projected/1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5-kube-api-access-t6zvz\") pod \"cluster-proxy-proxy-agent-7bb4687d54-bgcmw\" (UID: \"1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw"
Sep 08 20:26:47.728672 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.728646    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-lghl7"
Sep 08 20:26:47.747985 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.747961    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d998b5f7d-6bl45"
Sep 08 20:26:47.756715 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:47.756695    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw"
Sep 08 20:26:48.031141 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:48.031063    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-lghl7"]
Sep 08 20:26:48.033754 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:26:48.033728    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa62b2d_92ab_4958_83a4_5fa1c56ea4cd.slice/crio-62b5364ab84fae073dfc16884381f821629b58647d113575bc780e69ba2a62bc WatchSource:0}: Error finding container 62b5364ab84fae073dfc16884381f821629b58647d113575bc780e69ba2a62bc: Status 404 returned error can't find the container with id 62b5364ab84fae073dfc16884381f821629b58647d113575bc780e69ba2a62bc
Sep 08 20:26:48.056070 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:48.055956    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d998b5f7d-6bl45"]
Sep 08 20:26:48.058165 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:26:48.058142    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2393af3f_39b6_481d_bb56_5333012ace73.slice/crio-d155a0de25523e572ce99724b68a4bc018e3840338add57fd43b187a0aaf9ed8 WatchSource:0}: Error finding container d155a0de25523e572ce99724b68a4bc018e3840338add57fd43b187a0aaf9ed8: Status 404 returned error can't find the container with id d155a0de25523e572ce99724b68a4bc018e3840338add57fd43b187a0aaf9ed8
Sep 08 20:26:48.117125 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:26:48.117102    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bab2ef0_ee83_4b8f_9b0e_e20d368cbed5.slice/crio-97c70343428d6cc4608158c7f385247abb855d9de6a8a7c6bfea34115c099260 WatchSource:0}: Error finding container 97c70343428d6cc4608158c7f385247abb855d9de6a8a7c6bfea34115c099260: Status 404 returned error can't find the container with id 97c70343428d6cc4608158c7f385247abb855d9de6a8a7c6bfea34115c099260
Sep 08 20:26:48.125512 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:48.125486    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw"]
Sep 08 20:26:48.611074 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:48.611036    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw" event={"ID":"1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5","Type":"ContainerStarted","Data":"97c70343428d6cc4608158c7f385247abb855d9de6a8a7c6bfea34115c099260"}
Sep 08 20:26:48.612300 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:48.612274    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-lghl7" event={"ID":"3fa62b2d-92ab-4958-83a4-5fa1c56ea4cd","Type":"ContainerStarted","Data":"62b5364ab84fae073dfc16884381f821629b58647d113575bc780e69ba2a62bc"}
Sep 08 20:26:48.613680 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:48.613655    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d998b5f7d-6bl45" event={"ID":"2393af3f-39b6-481d-bb56-5333012ace73","Type":"ContainerStarted","Data":"d155a0de25523e572ce99724b68a4bc018e3840338add57fd43b187a0aaf9ed8"}
Sep 08 20:26:50.623634 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:50.623595    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-lghl7" event={"ID":"3fa62b2d-92ab-4958-83a4-5fa1c56ea4cd","Type":"ContainerStarted","Data":"1bebeb540f933a5582620cfa1f3dbaab3c47bed1b4f10bb2cd3032793bc54df1"}
Sep 08 20:26:50.624105 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:50.623662    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-lghl7" event={"ID":"3fa62b2d-92ab-4958-83a4-5fa1c56ea4cd","Type":"ContainerStarted","Data":"20f757350135c163f305c5965dbbe8a9e44d8a64a334870eb2c1d32d6c62b3b3"}
Sep 08 20:26:50.639541 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:50.639495    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-lghl7" podStartSLOduration=2.038182392 podStartE2EDuration="3.639477231s" totalImagesPullingTime="1.601294839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:26:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:26:50.638099927 +0000 UTC m=+63.889581285" watchObservedRunningTime="2026-09-08 20:26:50.639477231 +0000 UTC m=+63.890958589"
Sep 08 20:26:51.599293 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:51.599264    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pfxtz"
Sep 08 20:26:52.632433 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:52.632388    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d998b5f7d-6bl45" event={"ID":"2393af3f-39b6-481d-bb56-5333012ace73","Type":"ContainerStarted","Data":"9eef0090d7ad3d65cf3da445a24297631d2c411a3f39b3f8aae865ac948f4b22"}
Sep 08 20:26:52.634414 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:52.634386    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw" event={"ID":"1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5","Type":"ContainerStarted","Data":"6cba9cdcf039c65031c4286d317c11b460624ded80a1ec9b01f179c200af52a5"}
Sep 08 20:26:52.634532 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:52.634424    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw" event={"ID":"1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5","Type":"ContainerStarted","Data":"201a2a28277de4d629b0c65f0ad0ae0a5e47a436249312baef88481ec311f9c9"}
Sep 08 20:26:52.634532 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:52.634439    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw" event={"ID":"1bab2ef0-ee83-4b8f-9b0e-e20d368cbed5","Type":"ContainerStarted","Data":"bbb00f26df80d51b63ee6d86ac300ff86482c683a7e9ff827e24b13bf28a928b"}
Sep 08 20:26:52.652398 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:52.652354    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d998b5f7d-6bl45" podStartSLOduration=1.6069261620000002 podStartE2EDuration="5.652340226s" totalImagesPullingTime="4.045414064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:26:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:26:52.651356317 +0000 UTC m=+65.902837689" watchObservedRunningTime="2026-09-08 20:26:52.652340226 +0000 UTC m=+65.903821584"
Sep 08 20:26:52.671959 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:52.671857    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bb4687d54-bgcmw" podStartSLOduration=1.675254325 podStartE2EDuration="5.671838801s" totalImagesPullingTime="3.996584476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:26:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:26:52.671557762 +0000 UTC m=+65.923039119" watchObservedRunningTime="2026-09-08 20:26:52.671838801 +0000 UTC m=+65.923320158"
Sep 08 20:26:52.904043 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:52.903917    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs\") pod \"network-metrics-daemon-zwvng\" (UID: \"0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059\") " pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:26:52.906180 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:52.906160    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 20:26:52.916833 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:52.916807    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059-metrics-certs\") pod \"network-metrics-daemon-zwvng\" (UID: \"0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059\") " pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:26:52.934842 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:52.934818    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kbhtv\""
Sep 08 20:26:52.943688 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:52.943668    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwvng"
Sep 08 20:26:53.004827 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:53.004791    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ttj\" (UniqueName: \"kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj\") pod \"network-check-target-g8mj2\" (UID: \"cb9818b0-76cd-4be1-9c9e-7d47e0cb1607\") " pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:26:53.005002 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:53.004844    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret\") pod \"global-pull-secret-syncer-f5v7c\" (UID: \"0dc52e01-2095-40dd-8603-0fdf9d6ff1aa\") " pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:26:53.007266 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:53.007190    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 20:26:53.007266 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:53.007190    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 20:26:53.017986 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:53.017954    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 20:26:53.018185 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:53.018166    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0dc52e01-2095-40dd-8603-0fdf9d6ff1aa-original-pull-secret\") pod \"global-pull-secret-syncer-f5v7c\" (UID: \"0dc52e01-2095-40dd-8603-0fdf9d6ff1aa\") " pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:26:53.028828 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:53.028799    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6ttj\" (UniqueName: \"kubernetes.io/projected/cb9818b0-76cd-4be1-9c9e-7d47e0cb1607-kube-api-access-x6ttj\") pod \"network-check-target-g8mj2\" (UID: \"cb9818b0-76cd-4be1-9c9e-7d47e0cb1607\") " pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:26:53.209451 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:53.209372    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zwvng"]
Sep 08 20:26:53.212565 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:26:53.212528    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f58e4f0_1c4d_4ac4_8e36_f3b1cf7e5059.slice/crio-a4cd4b0ac619b9a775bcc6f9cc4636aa11ce66d93a0de93f78f8b2e6018be277 WatchSource:0}: Error finding container a4cd4b0ac619b9a775bcc6f9cc4636aa11ce66d93a0de93f78f8b2e6018be277: Status 404 returned error can't find the container with id a4cd4b0ac619b9a775bcc6f9cc4636aa11ce66d93a0de93f78f8b2e6018be277
Sep 08 20:26:53.243960 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:53.243937    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2nzg7\""
Sep 08 20:26:53.246728 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:53.246714    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5v7c"
Sep 08 20:26:53.252186 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:53.252163    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:26:53.518489 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:53.518419    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f5v7c"]
Sep 08 20:26:53.521375 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:26:53.521350    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc52e01_2095_40dd_8603_0fdf9d6ff1aa.slice/crio-8bd0462dd27e251d462ea9f7a4f293376eea9f8370cb463105bdc5c355c876c6 WatchSource:0}: Error finding container 8bd0462dd27e251d462ea9f7a4f293376eea9f8370cb463105bdc5c355c876c6: Status 404 returned error can't find the container with id 8bd0462dd27e251d462ea9f7a4f293376eea9f8370cb463105bdc5c355c876c6
Sep 08 20:26:53.525897 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:53.525867    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g8mj2"]
Sep 08 20:26:53.529301 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:26:53.529272    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9818b0_76cd_4be1_9c9e_7d47e0cb1607.slice/crio-324b8ce31af1f5c0891857f15ddce64671ad68674e567c3d5ec2a770738ce0c3 WatchSource:0}: Error finding container 324b8ce31af1f5c0891857f15ddce64671ad68674e567c3d5ec2a770738ce0c3: Status 404 returned error can't find the container with id 324b8ce31af1f5c0891857f15ddce64671ad68674e567c3d5ec2a770738ce0c3
Sep 08 20:26:53.639003 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:53.638967    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f5v7c" event={"ID":"0dc52e01-2095-40dd-8603-0fdf9d6ff1aa","Type":"ContainerStarted","Data":"8bd0462dd27e251d462ea9f7a4f293376eea9f8370cb463105bdc5c355c876c6"}
Sep 08 20:26:53.640465 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:53.640432    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zwvng" event={"ID":"0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059","Type":"ContainerStarted","Data":"a4cd4b0ac619b9a775bcc6f9cc4636aa11ce66d93a0de93f78f8b2e6018be277"}
Sep 08 20:26:53.641764 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:53.641737    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g8mj2" event={"ID":"cb9818b0-76cd-4be1-9c9e-7d47e0cb1607","Type":"ContainerStarted","Data":"324b8ce31af1f5c0891857f15ddce64671ad68674e567c3d5ec2a770738ce0c3"}
Sep 08 20:26:54.649232 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:54.649201    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zwvng" event={"ID":"0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059","Type":"ContainerStarted","Data":"c6b3d344f04f4b68e35b3837d655afbedcdf2fd499567865231acbac9bd88829"}
Sep 08 20:26:54.649637 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:54.649280    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zwvng" event={"ID":"0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059","Type":"ContainerStarted","Data":"42c54a79604b9adffd6cc82e2bf0628975f3df32feb783b8ad9eb379100dc7db"}
Sep 08 20:26:54.671704 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:54.671608    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zwvng" podStartSLOduration=66.748013009 podStartE2EDuration="1m7.67159339s" totalImagesPullingTime="923.580381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:26:54.670039063 +0000 UTC m=+67.921520422" watchObservedRunningTime="2026-09-08 20:26:54.67159339 +0000 UTC m=+67.923074747"
Sep 08 20:26:57.339574 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:57.339530    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/607dd30a-5e39-4cc8-91b0-cba0ad740873-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:57.342018 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:57.341989    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/607dd30a-5e39-4cc8-91b0-cba0ad740873-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kw8zt\" (UID: \"607dd30a-5e39-4cc8-91b0-cba0ad740873\") " pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:57.613398 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:57.613315    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4gd8m\""
Sep 08 20:26:57.621321 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:57.621297    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kw8zt"
Sep 08 20:26:59.319087 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:59.319050    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kw8zt"]
Sep 08 20:26:59.322506 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:26:59.322472    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod607dd30a_5e39_4cc8_91b0_cba0ad740873.slice/crio-182f5c0f40e8e519b0a1368664491c4398f1caab6931919d4a118dd5a880617d WatchSource:0}: Error finding container 182f5c0f40e8e519b0a1368664491c4398f1caab6931919d4a118dd5a880617d: Status 404 returned error can't find the container with id 182f5c0f40e8e519b0a1368664491c4398f1caab6931919d4a118dd5a880617d
Sep 08 20:26:59.668137 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:59.668100    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f5v7c" event={"ID":"0dc52e01-2095-40dd-8603-0fdf9d6ff1aa","Type":"ContainerStarted","Data":"25b57e9d073f15420845006a6c76a1fec789f71f29e2ee53ea485ea9a56d9968"}
Sep 08 20:26:59.669616 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:59.669591    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kw8zt" event={"ID":"607dd30a-5e39-4cc8-91b0-cba0ad740873","Type":"ContainerStarted","Data":"ca025ce9f345b5eac6a89b4d21ffc650396f2240da840126a734dbec330d645b"}
Sep 08 20:26:59.669741 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:59.669641    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kw8zt" event={"ID":"607dd30a-5e39-4cc8-91b0-cba0ad740873","Type":"ContainerStarted","Data":"182f5c0f40e8e519b0a1368664491c4398f1caab6931919d4a118dd5a880617d"}
Sep 08 20:26:59.670973 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:59.670949    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g8mj2" event={"ID":"cb9818b0-76cd-4be1-9c9e-7d47e0cb1607","Type":"ContainerStarted","Data":"177c678bc0510c9c59caa9d551ea3b53aaca0dd433952aec9520c3cd74a051c4"}
Sep 08 20:26:59.671078 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:59.671070    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:26:59.684764 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:26:59.684727    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-f5v7c" podStartSLOduration=66.164888741 podStartE2EDuration="1m11.684714572s" totalImagesPullingTime="5.519825831s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 20:25:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:26:59.68468435 +0000 UTC m=+72.936165707" watchObservedRunningTime="2026-09-08 20:26:59.684714572 +0000 UTC m=+72.936195912"
Sep 08 20:27:00.677446 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:00.677411    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kw8zt" event={"ID":"607dd30a-5e39-4cc8-91b0-cba0ad740873","Type":"ContainerStarted","Data":"72318f5b3b8bc4c8789133a2b4ee694e5e745e97fcfcc169ffc7c49c2a7554ab"}
Sep 08 20:27:01.683107 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:01.683076    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kw8zt" event={"ID":"607dd30a-5e39-4cc8-91b0-cba0ad740873","Type":"ContainerStarted","Data":"13f51d472dea2bf98cb8bf87a47b19af274927637d9657a591d4281fe00bfab3"}
Sep 08 20:27:01.702224 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:01.702167    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g8mj2" podStartSLOduration=69.189510538 podStartE2EDuration="1m14.702153523s" totalImagesPullingTime="5.512642985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:26:59.700416587 +0000 UTC m=+72.951897944" watchObservedRunningTime="2026-09-08 20:27:01.702153523 +0000 UTC m=+74.953634920"
Sep 08 20:27:01.702535 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:01.702513    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kw8zt" podStartSLOduration=34.884453727 podStartE2EDuration="36.702507592s" totalImagesPullingTime="1.818053865s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 20:26:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:27:01.701861476 +0000 UTC m=+74.953342833" watchObservedRunningTime="2026-09-08 20:27:01.702507592 +0000 UTC m=+74.953988948"
Sep 08 20:27:10.217345 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.217194    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w94bw"]
Sep 08 20:27:10.219723 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.219703    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.224001 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.223973    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 20:27:10.224156 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.223982    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 20:27:10.224268 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.224227    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 20:27:10.224392 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.224373    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 20:27:10.224486 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.224467    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nwv9r\""
Sep 08 20:27:10.225029 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.225003    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 20:27:10.225163 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.225016    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 20:27:10.326254 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.326211    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2131df78-3591-4c7f-8b0d-4e3653471b83-node-exporter-tls\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.326408 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.326262    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2131df78-3591-4c7f-8b0d-4e3653471b83-metrics-client-ca\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.326408 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.326280    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bttws\" (UniqueName: \"kubernetes.io/projected/2131df78-3591-4c7f-8b0d-4e3653471b83-kube-api-access-bttws\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.326408 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.326338    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2131df78-3591-4c7f-8b0d-4e3653471b83-root\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.326408 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.326383    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2131df78-3591-4c7f-8b0d-4e3653471b83-sys\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.326543 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.326412    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2131df78-3591-4c7f-8b0d-4e3653471b83-node-exporter-textfile\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.326543 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.326456    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2131df78-3591-4c7f-8b0d-4e3653471b83-node-exporter-accelerators-collector-config\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.326543 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.326499    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2131df78-3591-4c7f-8b0d-4e3653471b83-node-exporter-wtmp\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.326543 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.326528    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2131df78-3591-4c7f-8b0d-4e3653471b83-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.427481 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.427450    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2131df78-3591-4c7f-8b0d-4e3653471b83-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.427481 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.427485    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2131df78-3591-4c7f-8b0d-4e3653471b83-node-exporter-tls\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.427719 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.427508    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2131df78-3591-4c7f-8b0d-4e3653471b83-metrics-client-ca\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.427719 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.427523    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bttws\" (UniqueName: \"kubernetes.io/projected/2131df78-3591-4c7f-8b0d-4e3653471b83-kube-api-access-bttws\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.427719 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.427542    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2131df78-3591-4c7f-8b0d-4e3653471b83-root\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.427719 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.427567    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2131df78-3591-4c7f-8b0d-4e3653471b83-sys\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.427719 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.427600    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2131df78-3591-4c7f-8b0d-4e3653471b83-node-exporter-textfile\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.427719 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.427620    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2131df78-3591-4c7f-8b0d-4e3653471b83-root\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.427719 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.427659    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2131df78-3591-4c7f-8b0d-4e3653471b83-sys\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.427719 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.427632    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2131df78-3591-4c7f-8b0d-4e3653471b83-node-exporter-accelerators-collector-config\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.428130 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.427743    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2131df78-3591-4c7f-8b0d-4e3653471b83-node-exporter-wtmp\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.428130 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.427902    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2131df78-3591-4c7f-8b0d-4e3653471b83-node-exporter-wtmp\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.428130 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.428093    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2131df78-3591-4c7f-8b0d-4e3653471b83-node-exporter-textfile\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.428360 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.428342    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2131df78-3591-4c7f-8b0d-4e3653471b83-node-exporter-accelerators-collector-config\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.428416 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.428384    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2131df78-3591-4c7f-8b0d-4e3653471b83-metrics-client-ca\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.429948 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.429926    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2131df78-3591-4c7f-8b0d-4e3653471b83-node-exporter-tls\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.429998 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.429956    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2131df78-3591-4c7f-8b0d-4e3653471b83-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.435279 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.435258    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bttws\" (UniqueName: \"kubernetes.io/projected/2131df78-3591-4c7f-8b0d-4e3653471b83-kube-api-access-bttws\") pod \"node-exporter-w94bw\" (UID: \"2131df78-3591-4c7f-8b0d-4e3653471b83\") " pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.529807 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.529731    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w94bw"
Sep 08 20:27:10.541337 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:27:10.541306    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2131df78_3591_4c7f_8b0d_4e3653471b83.slice/crio-fca13746c08c93038f5e2304b6f67b669a672696017acc2515e290ce55e960d1 WatchSource:0}: Error finding container fca13746c08c93038f5e2304b6f67b669a672696017acc2515e290ce55e960d1: Status 404 returned error can't find the container with id fca13746c08c93038f5e2304b6f67b669a672696017acc2515e290ce55e960d1
Sep 08 20:27:10.713912 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:10.713878    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w94bw" event={"ID":"2131df78-3591-4c7f-8b0d-4e3653471b83","Type":"ContainerStarted","Data":"fca13746c08c93038f5e2304b6f67b669a672696017acc2515e290ce55e960d1"}
Sep 08 20:27:12.722474 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:12.722440    2552 generic.go:356] "Generic (PLEG): container finished" podID="2131df78-3591-4c7f-8b0d-4e3653471b83" containerID="52c578c79ed65908e0d78b54810cc1ac69cc01d786c8a65402cbc484fb2fa5d7" exitCode=0
Sep 08 20:27:12.722837 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:12.722525    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w94bw" event={"ID":"2131df78-3591-4c7f-8b0d-4e3653471b83","Type":"ContainerDied","Data":"52c578c79ed65908e0d78b54810cc1ac69cc01d786c8a65402cbc484fb2fa5d7"}
Sep 08 20:27:13.728411 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:13.728369    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w94bw" event={"ID":"2131df78-3591-4c7f-8b0d-4e3653471b83","Type":"ContainerStarted","Data":"1d33b162af8b01fa70fe292af5343ca44d0865f032f4888cb9f8e7e2b254e859"}
Sep 08 20:27:13.728799 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:13.728435    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w94bw" event={"ID":"2131df78-3591-4c7f-8b0d-4e3653471b83","Type":"ContainerStarted","Data":"40546263f3c43eb1dcb67e1c7463c2def0f29786a13034c4c7f3bc59132c3c70"}
Sep 08 20:27:13.747119 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:13.747066    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w94bw" podStartSLOduration=2.439845314 podStartE2EDuration="3.74705256s" totalImagesPullingTime="1.244751255s" totalInitContainerRuntime="62.455991ms" isStatefulPod=true podCreationTimestamp="2026-09-08 20:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:27:13.746305894 +0000 UTC m=+86.997787251" watchObservedRunningTime="2026-09-08 20:27:13.74705256 +0000 UTC m=+86.998533917"
Sep 08 20:27:16.291903 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.291871    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 20:27:16.294490 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.294474    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.296685 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.296648    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 08 20:27:16.296685 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.296648    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 08 20:27:16.297097 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.297076    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 08 20:27:16.297254 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.297109    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 08 20:27:16.297418 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.297398    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 08 20:27:16.297635 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.297616    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 08 20:27:16.297735 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.297648    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6p5ns\""
Sep 08 20:27:16.297735 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.297682    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 08 20:27:16.298415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.298262    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 08 20:27:16.298415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.298272    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 08 20:27:16.298415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.298288    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 08 20:27:16.298415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.298327    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 08 20:27:16.298415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.298400    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 08 20:27:16.298659 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.298505    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 08 20:27:16.298659 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.298597    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 08 20:27:16.298659 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.298643    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-20gjf8ugi7k6h\""
Sep 08 20:27:16.300391 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.300372    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 08 20:27:16.311685 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.311664    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 20:27:16.373420 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.373390    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-web-config\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.373420 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.373421    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de56a525-9845-4b83-8c6f-488b1514803e-config-out\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.373654 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.373442    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.373654 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.373459    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/de56a525-9845-4b83-8c6f-488b1514803e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.373654 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.373495    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de56a525-9845-4b83-8c6f-488b1514803e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.373654 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.373511    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.373654 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.373526    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/de56a525-9845-4b83-8c6f-488b1514803e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.373654 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.373543    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.373654 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.373563    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/de56a525-9845-4b83-8c6f-488b1514803e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.373654 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.373618    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-config\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.373654 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.373648    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.374034 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.373667    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de56a525-9845-4b83-8c6f-488b1514803e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.374034 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.373705    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de56a525-9845-4b83-8c6f-488b1514803e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.374034 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.373727    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/de56a525-9845-4b83-8c6f-488b1514803e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.374034 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.373742    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.374034 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.373763    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.374034 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.373788    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de56a525-9845-4b83-8c6f-488b1514803e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.374034 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.373849    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de56a525-9845-4b83-8c6f-488b1514803e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.374034 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.373864    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64mf5\" (UniqueName: \"kubernetes.io/projected/de56a525-9845-4b83-8c6f-488b1514803e-kube-api-access-64mf5\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.374034 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.373881    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.474755 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.474718    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de56a525-9845-4b83-8c6f-488b1514803e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.474755 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.474758    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64mf5\" (UniqueName: \"kubernetes.io/projected/de56a525-9845-4b83-8c6f-488b1514803e-kube-api-access-64mf5\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.474996 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.474785    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.474996 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.474821    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-web-config\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.474996 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.474846    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de56a525-9845-4b83-8c6f-488b1514803e-config-out\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.474996 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.474875    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.474996 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.474900    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/de56a525-9845-4b83-8c6f-488b1514803e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.474996 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.474926    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de56a525-9845-4b83-8c6f-488b1514803e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.474996 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.474947    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.474996 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.474973    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/de56a525-9845-4b83-8c6f-488b1514803e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.475427 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.474998    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.475427 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.475022    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/de56a525-9845-4b83-8c6f-488b1514803e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.475427 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.475075    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-config\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.475427 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.475100    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.475427 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.475136    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de56a525-9845-4b83-8c6f-488b1514803e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.475427 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.475163    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de56a525-9845-4b83-8c6f-488b1514803e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.475427 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.475187    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/de56a525-9845-4b83-8c6f-488b1514803e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.475427 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.475209    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.475427 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.475257    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.475427 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.475281    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de56a525-9845-4b83-8c6f-488b1514803e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.475896 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.475681    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de56a525-9845-4b83-8c6f-488b1514803e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.475896 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.475856    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/de56a525-9845-4b83-8c6f-488b1514803e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.476263 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.476205    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/de56a525-9845-4b83-8c6f-488b1514803e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.476451 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.476421    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de56a525-9845-4b83-8c6f-488b1514803e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.476842 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.476814    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de56a525-9845-4b83-8c6f-488b1514803e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.476989 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.476950    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/de56a525-9845-4b83-8c6f-488b1514803e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.477093 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.477069    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de56a525-9845-4b83-8c6f-488b1514803e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.478539 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.478509    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.478721 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.478659    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de56a525-9845-4b83-8c6f-488b1514803e-config-out\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.478981 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.478956    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de56a525-9845-4b83-8c6f-488b1514803e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.479107 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.479079    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-web-config\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.480018 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.479970    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.480115 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.480044    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.480115 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.480050    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.480381 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.480358    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-config\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.480507 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.480482    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.480747 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.480727    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.481356 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.481331    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/de56a525-9845-4b83-8c6f-488b1514803e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.481442 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.481425    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/de56a525-9845-4b83-8c6f-488b1514803e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.483138 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.483115    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64mf5\" (UniqueName: \"kubernetes.io/projected/de56a525-9845-4b83-8c6f-488b1514803e-kube-api-access-64mf5\") pod \"prometheus-k8s-0\" (UID: \"de56a525-9845-4b83-8c6f-488b1514803e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.603665 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.603634    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:16.884404 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:16.884325    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 20:27:16.887064 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:27:16.887037    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde56a525_9845_4b83_8c6f_488b1514803e.slice/crio-1863f785bfc3d28820dfea8938c7fabf0e79e00581b9965361fe1b51d16e0455 WatchSource:0}: Error finding container 1863f785bfc3d28820dfea8938c7fabf0e79e00581b9965361fe1b51d16e0455: Status 404 returned error can't find the container with id 1863f785bfc3d28820dfea8938c7fabf0e79e00581b9965361fe1b51d16e0455
Sep 08 20:27:17.744305 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:17.744274    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de56a525-9845-4b83-8c6f-488b1514803e","Type":"ContainerStarted","Data":"1863f785bfc3d28820dfea8938c7fabf0e79e00581b9965361fe1b51d16e0455"}
Sep 08 20:27:18.749197 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:18.749159    2552 generic.go:356] "Generic (PLEG): container finished" podID="de56a525-9845-4b83-8c6f-488b1514803e" containerID="0122ca60f7db98930d5919743400ef593ed5cc4afe8b38112874a79b10a5f537" exitCode=0
Sep 08 20:27:18.749684 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:18.749261    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de56a525-9845-4b83-8c6f-488b1514803e","Type":"ContainerDied","Data":"0122ca60f7db98930d5919743400ef593ed5cc4afe8b38112874a79b10a5f537"}
Sep 08 20:27:22.768337 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:22.768300    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de56a525-9845-4b83-8c6f-488b1514803e","Type":"ContainerStarted","Data":"c12a589fe0938063f8974741a7f228e2c2b39ba10ba604691432c9de284f95b9"}
Sep 08 20:27:22.768734 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:22.768381    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de56a525-9845-4b83-8c6f-488b1514803e","Type":"ContainerStarted","Data":"439c525de8b3a27ac175d3674c659d63b92d6d1e106557cd292b8b40acea3188"}
Sep 08 20:27:25.782211 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:25.782176    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de56a525-9845-4b83-8c6f-488b1514803e","Type":"ContainerStarted","Data":"b155d573f9bb2b2723980e92c5bfc45810307fc71834f038b24a27ef93a836fc"}
Sep 08 20:27:25.782603 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:25.782255    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de56a525-9845-4b83-8c6f-488b1514803e","Type":"ContainerStarted","Data":"147bf6a79abfc50ee4effa688f6e527185f1b7e6686dd813bd2f7b9be5f00bfd"}
Sep 08 20:27:25.782603 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:25.782284    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de56a525-9845-4b83-8c6f-488b1514803e","Type":"ContainerStarted","Data":"a331f4dc019f92caab839de3d0aca038c231f1911b5e573d48ab90e5c095cc1d"}
Sep 08 20:27:25.782603 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:25.782312    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de56a525-9845-4b83-8c6f-488b1514803e","Type":"ContainerStarted","Data":"e7bfca16c0bb3005889023dcfb3682284c73eff1d1d7eb63fa30ec341dd6c28a"}
Sep 08 20:27:25.813406 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:25.813345    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.84509563 podStartE2EDuration="9.813324907s" totalImagesPullingTime="6.948920409s" totalInitContainerRuntime="19.308868ms" isStatefulPod=false podCreationTimestamp="2026-09-08 20:27:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:27:25.811702849 +0000 UTC m=+99.063184229" watchObservedRunningTime="2026-09-08 20:27:25.813324907 +0000 UTC m=+99.064806265"
Sep 08 20:27:26.603827 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:26.603786    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:27:30.680492 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:27:30.680460    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g8mj2"
Sep 08 20:28:16.603927 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:28:16.603878    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:28:16.622670 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:28:16.622637    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:28:16.985388 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:28:16.985317    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 20:30:47.098336 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:30:47.098300    2552 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 20:31:01.902716 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:01.902675    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl"]
Sep 08 20:31:01.906441 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:01.906426    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl"
Sep 08 20:31:01.908639 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:01.908616    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 08 20:31:01.909309 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:01.909285    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 08 20:31:01.909445 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:01.909424    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 08 20:31:01.909654 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:01.909638    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sv8zf\""
Sep 08 20:31:01.909737 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:01.909670    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 08 20:31:01.924847 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:01.924820    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl"]
Sep 08 20:31:02.028829 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:02.028802    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2b591c3f-1022-494d-8b19-eceefee547d0-webhook-cert\") pod \"opendatahub-operator-controller-manager-dbd9f8d48-6jpjl\" (UID: \"2b591c3f-1022-494d-8b19-eceefee547d0\") " pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl"
Sep 08 20:31:02.029005 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:02.028846    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b591c3f-1022-494d-8b19-eceefee547d0-tmp\") pod \"opendatahub-operator-controller-manager-dbd9f8d48-6jpjl\" (UID: \"2b591c3f-1022-494d-8b19-eceefee547d0\") " pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl"
Sep 08 20:31:02.029005 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:02.028866    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2b591c3f-1022-494d-8b19-eceefee547d0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dbd9f8d48-6jpjl\" (UID: \"2b591c3f-1022-494d-8b19-eceefee547d0\") " pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl"
Sep 08 20:31:02.029005 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:02.028882    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxs56\" (UniqueName: \"kubernetes.io/projected/2b591c3f-1022-494d-8b19-eceefee547d0-kube-api-access-bxs56\") pod \"opendatahub-operator-controller-manager-dbd9f8d48-6jpjl\" (UID: \"2b591c3f-1022-494d-8b19-eceefee547d0\") " pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl"
Sep 08 20:31:02.029005 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:02.028970    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2b591c3f-1022-494d-8b19-eceefee547d0-manifests\") pod \"opendatahub-operator-controller-manager-dbd9f8d48-6jpjl\" (UID: \"2b591c3f-1022-494d-8b19-eceefee547d0\") " pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl"
Sep 08 20:31:02.129353 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:02.129318    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2b591c3f-1022-494d-8b19-eceefee547d0-webhook-cert\") pod \"opendatahub-operator-controller-manager-dbd9f8d48-6jpjl\" (UID: \"2b591c3f-1022-494d-8b19-eceefee547d0\") " pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl"
Sep 08 20:31:02.129528 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:02.129367    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b591c3f-1022-494d-8b19-eceefee547d0-tmp\") pod \"opendatahub-operator-controller-manager-dbd9f8d48-6jpjl\" (UID: \"2b591c3f-1022-494d-8b19-eceefee547d0\") " pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl"
Sep 08 20:31:02.129528 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:02.129386    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2b591c3f-1022-494d-8b19-eceefee547d0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dbd9f8d48-6jpjl\" (UID: \"2b591c3f-1022-494d-8b19-eceefee547d0\") " pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl"
Sep 08 20:31:02.129528 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:02.129403    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxs56\" (UniqueName: \"kubernetes.io/projected/2b591c3f-1022-494d-8b19-eceefee547d0-kube-api-access-bxs56\") pod \"opendatahub-operator-controller-manager-dbd9f8d48-6jpjl\" (UID: \"2b591c3f-1022-494d-8b19-eceefee547d0\") " pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl"
Sep 08 20:31:02.129528 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:02.129439    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2b591c3f-1022-494d-8b19-eceefee547d0-manifests\") pod \"opendatahub-operator-controller-manager-dbd9f8d48-6jpjl\" (UID: \"2b591c3f-1022-494d-8b19-eceefee547d0\") " pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl"
Sep 08 20:31:02.129894 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:02.129866    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2b591c3f-1022-494d-8b19-eceefee547d0-manifests\") pod \"opendatahub-operator-controller-manager-dbd9f8d48-6jpjl\" (UID: \"2b591c3f-1022-494d-8b19-eceefee547d0\") " pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl"
Sep 08 20:31:02.131649 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:02.131621    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b591c3f-1022-494d-8b19-eceefee547d0-tmp\") pod \"opendatahub-operator-controller-manager-dbd9f8d48-6jpjl\" (UID: \"2b591c3f-1022-494d-8b19-eceefee547d0\") " pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl"
Sep 08 20:31:02.131770 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:02.131736    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2b591c3f-1022-494d-8b19-eceefee547d0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dbd9f8d48-6jpjl\" (UID: \"2b591c3f-1022-494d-8b19-eceefee547d0\") " pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl"
Sep 08 20:31:02.131845 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:02.131827    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2b591c3f-1022-494d-8b19-eceefee547d0-webhook-cert\") pod \"opendatahub-operator-controller-manager-dbd9f8d48-6jpjl\" (UID: \"2b591c3f-1022-494d-8b19-eceefee547d0\") " pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl"
Sep 08 20:31:02.137063 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:02.137039    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxs56\" (UniqueName: \"kubernetes.io/projected/2b591c3f-1022-494d-8b19-eceefee547d0-kube-api-access-bxs56\") pod \"opendatahub-operator-controller-manager-dbd9f8d48-6jpjl\" (UID: \"2b591c3f-1022-494d-8b19-eceefee547d0\") " pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl"
Sep 08 20:31:02.217260 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:02.217149    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl"
Sep 08 20:31:02.493128 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:02.493047    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl"]
Sep 08 20:31:02.496767 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:31:02.496739    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b591c3f_1022_494d_8b19_eceefee547d0.slice/crio-d8417d48d611dcff4f076dcaa2541f233b2ca84527406fca7383e204190a8206 WatchSource:0}: Error finding container d8417d48d611dcff4f076dcaa2541f233b2ca84527406fca7383e204190a8206: Status 404 returned error can't find the container with id d8417d48d611dcff4f076dcaa2541f233b2ca84527406fca7383e204190a8206
Sep 08 20:31:02.498718 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:02.498702    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 20:31:02.515143 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:02.515113    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl" event={"ID":"2b591c3f-1022-494d-8b19-eceefee547d0","Type":"ContainerStarted","Data":"d8417d48d611dcff4f076dcaa2541f233b2ca84527406fca7383e204190a8206"}
Sep 08 20:31:06.531883 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:06.531845    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl" event={"ID":"2b591c3f-1022-494d-8b19-eceefee547d0","Type":"ContainerStarted","Data":"687649bdc9bac37c4e67cb1a0e3d0b2705beb414a88419f0f69361c2324f6ccf"}
Sep 08 20:31:07.537163 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:07.537129    2552 generic.go:356] "Generic (PLEG): container finished" podID="2b591c3f-1022-494d-8b19-eceefee547d0" containerID="687649bdc9bac37c4e67cb1a0e3d0b2705beb414a88419f0f69361c2324f6ccf" exitCode=0
Sep 08 20:31:07.537552 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:07.537194    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl" event={"ID":"2b591c3f-1022-494d-8b19-eceefee547d0","Type":"ContainerDied","Data":"687649bdc9bac37c4e67cb1a0e3d0b2705beb414a88419f0f69361c2324f6ccf"}
Sep 08 20:31:08.542805 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:08.542771    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl" event={"ID":"2b591c3f-1022-494d-8b19-eceefee547d0","Type":"ContainerStarted","Data":"124e86b583094d6e46e3f3cf5f94dce7fca9e64eea280781befc3cd25fcc4623"}
Sep 08 20:31:08.543189 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:08.542904    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl"
Sep 08 20:31:08.567286 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:08.567221    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl" podStartSLOduration=3.187350837 podStartE2EDuration="7.567208093s" totalImagesPullingTime="3.369565535s" totalInitContainerRuntime="1.010291721s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:31:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:31:08.566221779 +0000 UTC m=+321.817703141" watchObservedRunningTime="2026-09-08 20:31:08.567208093 +0000 UTC m=+321.818689449"
Sep 08 20:31:19.550488 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:19.550456    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-dbd9f8d48-6jpjl"
Sep 08 20:31:38.455075 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:38.455037    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-57c49bb89d-qmlxh"]
Sep 08 20:31:38.463906 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:38.463883    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57c49bb89d-qmlxh"
Sep 08 20:31:38.467107 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:38.467081    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-42k6v\""
Sep 08 20:31:38.467259 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:38.467213    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 08 20:31:38.467259 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:38.467227    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 08 20:31:38.467385 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:38.467228    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 08 20:31:38.467385 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:38.467225    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 08 20:31:38.467385 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:38.467228    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 08 20:31:38.470099 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:38.470080    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57c49bb89d-qmlxh"]
Sep 08 20:31:38.513104 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:38.513074    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bf6a7f6-3bf9-4000-9eec-099d382568b8-cert\") pod \"lws-controller-manager-57c49bb89d-qmlxh\" (UID: \"0bf6a7f6-3bf9-4000-9eec-099d382568b8\") " pod="openshift-lws-operator/lws-controller-manager-57c49bb89d-qmlxh"
Sep 08 20:31:38.513264 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:38.513107    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0bf6a7f6-3bf9-4000-9eec-099d382568b8-metrics-cert\") pod \"lws-controller-manager-57c49bb89d-qmlxh\" (UID: \"0bf6a7f6-3bf9-4000-9eec-099d382568b8\") " pod="openshift-lws-operator/lws-controller-manager-57c49bb89d-qmlxh"
Sep 08 20:31:38.513264 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:38.513144    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0bf6a7f6-3bf9-4000-9eec-099d382568b8-manager-config\") pod \"lws-controller-manager-57c49bb89d-qmlxh\" (UID: \"0bf6a7f6-3bf9-4000-9eec-099d382568b8\") " pod="openshift-lws-operator/lws-controller-manager-57c49bb89d-qmlxh"
Sep 08 20:31:38.513264 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:38.513222    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86km5\" (UniqueName: \"kubernetes.io/projected/0bf6a7f6-3bf9-4000-9eec-099d382568b8-kube-api-access-86km5\") pod \"lws-controller-manager-57c49bb89d-qmlxh\" (UID: \"0bf6a7f6-3bf9-4000-9eec-099d382568b8\") " pod="openshift-lws-operator/lws-controller-manager-57c49bb89d-qmlxh"
Sep 08 20:31:38.613738 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:38.613700    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bf6a7f6-3bf9-4000-9eec-099d382568b8-cert\") pod \"lws-controller-manager-57c49bb89d-qmlxh\" (UID: \"0bf6a7f6-3bf9-4000-9eec-099d382568b8\") " pod="openshift-lws-operator/lws-controller-manager-57c49bb89d-qmlxh"
Sep 08 20:31:38.613928 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:38.613751    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0bf6a7f6-3bf9-4000-9eec-099d382568b8-metrics-cert\") pod \"lws-controller-manager-57c49bb89d-qmlxh\" (UID: \"0bf6a7f6-3bf9-4000-9eec-099d382568b8\") " pod="openshift-lws-operator/lws-controller-manager-57c49bb89d-qmlxh"
Sep 08 20:31:38.613928 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:38.613805    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0bf6a7f6-3bf9-4000-9eec-099d382568b8-manager-config\") pod \"lws-controller-manager-57c49bb89d-qmlxh\" (UID: \"0bf6a7f6-3bf9-4000-9eec-099d382568b8\") " pod="openshift-lws-operator/lws-controller-manager-57c49bb89d-qmlxh"
Sep 08 20:31:38.613928 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:38.613839    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86km5\" (UniqueName: \"kubernetes.io/projected/0bf6a7f6-3bf9-4000-9eec-099d382568b8-kube-api-access-86km5\") pod \"lws-controller-manager-57c49bb89d-qmlxh\" (UID: \"0bf6a7f6-3bf9-4000-9eec-099d382568b8\") " pod="openshift-lws-operator/lws-controller-manager-57c49bb89d-qmlxh"
Sep 08 20:31:38.614629 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:38.614602    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0bf6a7f6-3bf9-4000-9eec-099d382568b8-manager-config\") pod \"lws-controller-manager-57c49bb89d-qmlxh\" (UID: \"0bf6a7f6-3bf9-4000-9eec-099d382568b8\") " pod="openshift-lws-operator/lws-controller-manager-57c49bb89d-qmlxh"
Sep 08 20:31:38.616549 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:38.616513    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0bf6a7f6-3bf9-4000-9eec-099d382568b8-metrics-cert\") pod \"lws-controller-manager-57c49bb89d-qmlxh\" (UID: \"0bf6a7f6-3bf9-4000-9eec-099d382568b8\") " pod="openshift-lws-operator/lws-controller-manager-57c49bb89d-qmlxh"
Sep 08 20:31:38.616667 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:38.616605    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bf6a7f6-3bf9-4000-9eec-099d382568b8-cert\") pod \"lws-controller-manager-57c49bb89d-qmlxh\" (UID: \"0bf6a7f6-3bf9-4000-9eec-099d382568b8\") " pod="openshift-lws-operator/lws-controller-manager-57c49bb89d-qmlxh"
Sep 08 20:31:38.622886 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:38.622862    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86km5\" (UniqueName: \"kubernetes.io/projected/0bf6a7f6-3bf9-4000-9eec-099d382568b8-kube-api-access-86km5\") pod \"lws-controller-manager-57c49bb89d-qmlxh\" (UID: \"0bf6a7f6-3bf9-4000-9eec-099d382568b8\") " pod="openshift-lws-operator/lws-controller-manager-57c49bb89d-qmlxh"
Sep 08 20:31:38.773976 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:38.773894    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57c49bb89d-qmlxh"
Sep 08 20:31:39.062392 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.062368    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57c49bb89d-qmlxh"]
Sep 08 20:31:39.064724 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:31:39.064698    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf6a7f6_3bf9_4000_9eec_099d382568b8.slice/crio-62e74f35152a042fd8f88cffb7748d2aa626e28912bbb1f0a2ee37157eb09362 WatchSource:0}: Error finding container 62e74f35152a042fd8f88cffb7748d2aa626e28912bbb1f0a2ee37157eb09362: Status 404 returned error can't find the container with id 62e74f35152a042fd8f88cffb7748d2aa626e28912bbb1f0a2ee37157eb09362
Sep 08 20:31:39.575510 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.575479    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"]
Sep 08 20:31:39.578610 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.578592    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:39.581819 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.581798    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 08 20:31:39.582429 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.582411    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 08 20:31:39.582780 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.582759    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 08 20:31:39.582897 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.582872    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rctrw\""
Sep 08 20:31:39.583013 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.582927    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 20:31:39.583013 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.582963    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 08 20:31:39.583177 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.583163    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 20:31:39.617396 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.617364    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"]
Sep 08 20:31:39.620074 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.620044    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1d416b-0cc0-4214-9d2b-83d341f0faee-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-nbq9q\" (UID: \"ac1d416b-0cc0-4214-9d2b-83d341f0faee\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:39.620196 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.620095    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmpbl\" (UniqueName: \"kubernetes.io/projected/ac1d416b-0cc0-4214-9d2b-83d341f0faee-kube-api-access-kmpbl\") pod \"istiod-openshift-gateway-5677ccb49b-nbq9q\" (UID: \"ac1d416b-0cc0-4214-9d2b-83d341f0faee\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:39.620196 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.620115    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1d416b-0cc0-4214-9d2b-83d341f0faee-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-nbq9q\" (UID: \"ac1d416b-0cc0-4214-9d2b-83d341f0faee\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:39.620196 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.620150    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac1d416b-0cc0-4214-9d2b-83d341f0faee-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-nbq9q\" (UID: \"ac1d416b-0cc0-4214-9d2b-83d341f0faee\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:39.620196 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.620186    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac1d416b-0cc0-4214-9d2b-83d341f0faee-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-nbq9q\" (UID: \"ac1d416b-0cc0-4214-9d2b-83d341f0faee\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:39.620428 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.620253    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac1d416b-0cc0-4214-9d2b-83d341f0faee-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-nbq9q\" (UID: \"ac1d416b-0cc0-4214-9d2b-83d341f0faee\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:39.620428 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.620282    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac1d416b-0cc0-4214-9d2b-83d341f0faee-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-nbq9q\" (UID: \"ac1d416b-0cc0-4214-9d2b-83d341f0faee\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:39.660479 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.660444    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57c49bb89d-qmlxh" event={"ID":"0bf6a7f6-3bf9-4000-9eec-099d382568b8","Type":"ContainerStarted","Data":"62e74f35152a042fd8f88cffb7748d2aa626e28912bbb1f0a2ee37157eb09362"}
Sep 08 20:31:39.720936 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.720906    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac1d416b-0cc0-4214-9d2b-83d341f0faee-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-nbq9q\" (UID: \"ac1d416b-0cc0-4214-9d2b-83d341f0faee\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:39.721091 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.720942    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac1d416b-0cc0-4214-9d2b-83d341f0faee-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-nbq9q\" (UID: \"ac1d416b-0cc0-4214-9d2b-83d341f0faee\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:39.721091 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.720971    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1d416b-0cc0-4214-9d2b-83d341f0faee-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-nbq9q\" (UID: \"ac1d416b-0cc0-4214-9d2b-83d341f0faee\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:39.721268 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.721092    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmpbl\" (UniqueName: \"kubernetes.io/projected/ac1d416b-0cc0-4214-9d2b-83d341f0faee-kube-api-access-kmpbl\") pod \"istiod-openshift-gateway-5677ccb49b-nbq9q\" (UID: \"ac1d416b-0cc0-4214-9d2b-83d341f0faee\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:39.721268 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.721129    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1d416b-0cc0-4214-9d2b-83d341f0faee-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-nbq9q\" (UID: \"ac1d416b-0cc0-4214-9d2b-83d341f0faee\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:39.721268 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.721164    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac1d416b-0cc0-4214-9d2b-83d341f0faee-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-nbq9q\" (UID: \"ac1d416b-0cc0-4214-9d2b-83d341f0faee\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:39.721268 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.721213    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac1d416b-0cc0-4214-9d2b-83d341f0faee-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-nbq9q\" (UID: \"ac1d416b-0cc0-4214-9d2b-83d341f0faee\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:39.722172 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.722141    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac1d416b-0cc0-4214-9d2b-83d341f0faee-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-nbq9q\" (UID: \"ac1d416b-0cc0-4214-9d2b-83d341f0faee\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:39.723332 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.723312    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1d416b-0cc0-4214-9d2b-83d341f0faee-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-nbq9q\" (UID: \"ac1d416b-0cc0-4214-9d2b-83d341f0faee\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:39.723479 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.723463    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac1d416b-0cc0-4214-9d2b-83d341f0faee-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-nbq9q\" (UID: \"ac1d416b-0cc0-4214-9d2b-83d341f0faee\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:39.723561 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.723542    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac1d416b-0cc0-4214-9d2b-83d341f0faee-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-nbq9q\" (UID: \"ac1d416b-0cc0-4214-9d2b-83d341f0faee\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:39.723648 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.723633    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac1d416b-0cc0-4214-9d2b-83d341f0faee-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-nbq9q\" (UID: \"ac1d416b-0cc0-4214-9d2b-83d341f0faee\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:39.729405 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.729382    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmpbl\" (UniqueName: \"kubernetes.io/projected/ac1d416b-0cc0-4214-9d2b-83d341f0faee-kube-api-access-kmpbl\") pod \"istiod-openshift-gateway-5677ccb49b-nbq9q\" (UID: \"ac1d416b-0cc0-4214-9d2b-83d341f0faee\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:39.729490 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.729457    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1d416b-0cc0-4214-9d2b-83d341f0faee-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-nbq9q\" (UID: \"ac1d416b-0cc0-4214-9d2b-83d341f0faee\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:39.887804 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:39.887768    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:40.184820 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:40.184636    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"]
Sep 08 20:31:40.188485 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:31:40.188293    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1d416b_0cc0_4214_9d2b_83d341f0faee.slice/crio-44eae939899d225acffdfb081d06402dea6c239093bf066628a47f52df6d42a3 WatchSource:0}: Error finding container 44eae939899d225acffdfb081d06402dea6c239093bf066628a47f52df6d42a3: Status 404 returned error can't find the container with id 44eae939899d225acffdfb081d06402dea6c239093bf066628a47f52df6d42a3
Sep 08 20:31:40.665920 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:40.665889    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q" event={"ID":"ac1d416b-0cc0-4214-9d2b-83d341f0faee","Type":"ContainerStarted","Data":"44eae939899d225acffdfb081d06402dea6c239093bf066628a47f52df6d42a3"}
Sep 08 20:31:42.684843 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:42.683987    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57c49bb89d-qmlxh" event={"ID":"0bf6a7f6-3bf9-4000-9eec-099d382568b8","Type":"ContainerStarted","Data":"9ef6ddec1785f978b336dfc7fc8b7ec91ee38a74ae8d7d402b4138e4a7d87cc4"}
Sep 08 20:31:42.684843 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:42.684692    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-57c49bb89d-qmlxh"
Sep 08 20:31:42.712059 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:42.712005    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-57c49bb89d-qmlxh" podStartSLOduration=1.9981272460000001 podStartE2EDuration="4.711983164s" totalImagesPullingTime="2.713855918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:31:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:31:42.710854801 +0000 UTC m=+355.962336161" watchObservedRunningTime="2026-09-08 20:31:42.711983164 +0000 UTC m=+355.963464523"
Sep 08 20:31:43.573467 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:43.573411    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 08 20:31:43.573537 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:43.573504    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 08 20:31:43.689377 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:43.689333    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q" event={"ID":"ac1d416b-0cc0-4214-9d2b-83d341f0faee","Type":"ContainerStarted","Data":"31690f4774bf1e42849884e406cef32dfc27a0c707a2662605d1f173a79cca30"}
Sep 08 20:31:43.747100 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:43.747045    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q" podStartSLOduration=1.364677675 podStartE2EDuration="4.747027225s" totalImagesPullingTime="3.38234955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:31:43.74606963 +0000 UTC m=+356.997550988" watchObservedRunningTime="2026-09-08 20:31:43.747027225 +0000 UTC m=+356.998508582"
Sep 08 20:31:44.694548 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:44.694498    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:44.696715 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:44.696678    2552 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-nbq9q container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 08 20:31:44.696875 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:44.696742    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q" podUID="ac1d416b-0cc0-4214-9d2b-83d341f0faee" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 20:31:45.698877 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:45.698849    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbq9q"
Sep 08 20:31:54.697042 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:31:54.697007    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-57c49bb89d-qmlxh"
Sep 08 20:33:45.523762 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:33:45.523728    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:33:45.525785 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:33:45.525769    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-2t78w"
Sep 08 20:33:45.528055 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:33:45.528025    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 08 20:33:45.528180 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:33:45.528069    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 08 20:33:45.528706 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:33:45.528679    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 08 20:33:45.528706 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:33:45.528689    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-j2hc8\""
Sep 08 20:33:45.537059 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:33:45.537038    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:33:45.551041 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:33:45.551017    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:33:45.622616 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:33:45.622585    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rdjg\" (UniqueName: \"kubernetes.io/projected/0149abc2-b2c6-4bb7-b05c-5a7aeed704a9-kube-api-access-5rdjg\") pod \"limitador-limitador-84994f7ffc-2t78w\" (UID: \"0149abc2-b2c6-4bb7-b05c-5a7aeed704a9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2t78w"
Sep 08 20:33:45.622809 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:33:45.622627    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0149abc2-b2c6-4bb7-b05c-5a7aeed704a9-config-file\") pod \"limitador-limitador-84994f7ffc-2t78w\" (UID: \"0149abc2-b2c6-4bb7-b05c-5a7aeed704a9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2t78w"
Sep 08 20:33:45.723278 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:33:45.723227    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0149abc2-b2c6-4bb7-b05c-5a7aeed704a9-config-file\") pod \"limitador-limitador-84994f7ffc-2t78w\" (UID: \"0149abc2-b2c6-4bb7-b05c-5a7aeed704a9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2t78w"
Sep 08 20:33:45.723436 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:33:45.723334    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rdjg\" (UniqueName: \"kubernetes.io/projected/0149abc2-b2c6-4bb7-b05c-5a7aeed704a9-kube-api-access-5rdjg\") pod \"limitador-limitador-84994f7ffc-2t78w\" (UID: \"0149abc2-b2c6-4bb7-b05c-5a7aeed704a9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2t78w"
Sep 08 20:33:45.723829 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:33:45.723808    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0149abc2-b2c6-4bb7-b05c-5a7aeed704a9-config-file\") pod \"limitador-limitador-84994f7ffc-2t78w\" (UID: \"0149abc2-b2c6-4bb7-b05c-5a7aeed704a9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2t78w"
Sep 08 20:33:45.731761 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:33:45.731732    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rdjg\" (UniqueName: \"kubernetes.io/projected/0149abc2-b2c6-4bb7-b05c-5a7aeed704a9-kube-api-access-5rdjg\") pod \"limitador-limitador-84994f7ffc-2t78w\" (UID: \"0149abc2-b2c6-4bb7-b05c-5a7aeed704a9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2t78w"
Sep 08 20:33:45.835618 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:33:45.835587    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-2t78w"
Sep 08 20:33:46.104445 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:33:46.104420    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:33:46.106719 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:33:46.106697    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0149abc2_b2c6_4bb7_b05c_5a7aeed704a9.slice/crio-753eecdc992adf6d06bb009fb63cf32608063cb270e5f6594b26f6554e7d99cf WatchSource:0}: Error finding container 753eecdc992adf6d06bb009fb63cf32608063cb270e5f6594b26f6554e7d99cf: Status 404 returned error can't find the container with id 753eecdc992adf6d06bb009fb63cf32608063cb270e5f6594b26f6554e7d99cf
Sep 08 20:33:46.130814 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:33:46.130790    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-2t78w" event={"ID":"0149abc2-b2c6-4bb7-b05c-5a7aeed704a9","Type":"ContainerStarted","Data":"753eecdc992adf6d06bb009fb63cf32608063cb270e5f6594b26f6554e7d99cf"}
Sep 08 20:33:49.144149 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:33:49.144108    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-2t78w" event={"ID":"0149abc2-b2c6-4bb7-b05c-5a7aeed704a9","Type":"ContainerStarted","Data":"82b6350d4cd382fdb2b3b434e1ae540be1a66efe9b7db40a0146e786d51f8dcf"}
Sep 08 20:33:49.144583 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:33:49.144261    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-2t78w"
Sep 08 20:33:49.165502 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:33:49.165449    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-2t78w" podStartSLOduration=2.041130054 podStartE2EDuration="4.165434205s" totalImagesPullingTime="2.124304151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:33:49.163979698 +0000 UTC m=+482.415461055" watchObservedRunningTime="2026-09-08 20:33:49.165434205 +0000 UTC m=+482.416915561"
Sep 08 20:34:00.149834 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:34:00.149801    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-2t78w"
Sep 08 20:35:00.803209 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:00.803132    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:35:16.316600 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.316568    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"]
Sep 08 20:35:16.319717 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.319702    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:16.322131 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.322105    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xsfxp\""
Sep 08 20:35:16.322300 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.322130    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 08 20:35:16.322856 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.322838    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 08 20:35:16.322946 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.322930    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 08 20:35:16.329964 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.329945    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"]
Sep 08 20:35:16.435398 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.435365    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc17884e-e7e5-4760-8012-871ab8238e71-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7\" (UID: \"bc17884e-e7e5-4760-8012-871ab8238e71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:16.435398 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.435398    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc17884e-e7e5-4760-8012-871ab8238e71-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7\" (UID: \"bc17884e-e7e5-4760-8012-871ab8238e71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:16.435638 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.435424    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc17884e-e7e5-4760-8012-871ab8238e71-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7\" (UID: \"bc17884e-e7e5-4760-8012-871ab8238e71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:16.435638 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.435540    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc17884e-e7e5-4760-8012-871ab8238e71-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7\" (UID: \"bc17884e-e7e5-4760-8012-871ab8238e71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:16.435638 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.435614    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rplmm\" (UniqueName: \"kubernetes.io/projected/bc17884e-e7e5-4760-8012-871ab8238e71-kube-api-access-rplmm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7\" (UID: \"bc17884e-e7e5-4760-8012-871ab8238e71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:16.435775 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.435655    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc17884e-e7e5-4760-8012-871ab8238e71-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7\" (UID: \"bc17884e-e7e5-4760-8012-871ab8238e71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:16.536870 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.536839    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rplmm\" (UniqueName: \"kubernetes.io/projected/bc17884e-e7e5-4760-8012-871ab8238e71-kube-api-access-rplmm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7\" (UID: \"bc17884e-e7e5-4760-8012-871ab8238e71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:16.537008 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.536881    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc17884e-e7e5-4760-8012-871ab8238e71-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7\" (UID: \"bc17884e-e7e5-4760-8012-871ab8238e71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:16.537008 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.536914    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc17884e-e7e5-4760-8012-871ab8238e71-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7\" (UID: \"bc17884e-e7e5-4760-8012-871ab8238e71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:16.537008 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.536932    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc17884e-e7e5-4760-8012-871ab8238e71-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7\" (UID: \"bc17884e-e7e5-4760-8012-871ab8238e71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:16.537008 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.536948    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc17884e-e7e5-4760-8012-871ab8238e71-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7\" (UID: \"bc17884e-e7e5-4760-8012-871ab8238e71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:16.537201 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.537123    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc17884e-e7e5-4760-8012-871ab8238e71-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7\" (UID: \"bc17884e-e7e5-4760-8012-871ab8238e71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:16.537352 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.537334    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc17884e-e7e5-4760-8012-871ab8238e71-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7\" (UID: \"bc17884e-e7e5-4760-8012-871ab8238e71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:16.537437 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.537418    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc17884e-e7e5-4760-8012-871ab8238e71-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7\" (UID: \"bc17884e-e7e5-4760-8012-871ab8238e71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:16.537485 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.537469    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc17884e-e7e5-4760-8012-871ab8238e71-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7\" (UID: \"bc17884e-e7e5-4760-8012-871ab8238e71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:16.539221 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.539203    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc17884e-e7e5-4760-8012-871ab8238e71-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7\" (UID: \"bc17884e-e7e5-4760-8012-871ab8238e71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:16.539406 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.539388    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc17884e-e7e5-4760-8012-871ab8238e71-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7\" (UID: \"bc17884e-e7e5-4760-8012-871ab8238e71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:16.544367 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.544346    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rplmm\" (UniqueName: \"kubernetes.io/projected/bc17884e-e7e5-4760-8012-871ab8238e71-kube-api-access-rplmm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7\" (UID: \"bc17884e-e7e5-4760-8012-871ab8238e71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:16.630487 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.630460    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:16.800889 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.800856    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:35:16.898444 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:16.898420    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"]
Sep 08 20:35:16.900624 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:35:16.900597    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc17884e_e7e5_4760_8012_871ab8238e71.slice/crio-2125ded690225bd39043856f269868378ae5cab7dc7586ea99e393235aabe6fd WatchSource:0}: Error finding container 2125ded690225bd39043856f269868378ae5cab7dc7586ea99e393235aabe6fd: Status 404 returned error can't find the container with id 2125ded690225bd39043856f269868378ae5cab7dc7586ea99e393235aabe6fd
Sep 08 20:35:17.472191 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:17.472161    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7" event={"ID":"bc17884e-e7e5-4760-8012-871ab8238e71","Type":"ContainerStarted","Data":"2125ded690225bd39043856f269868378ae5cab7dc7586ea99e393235aabe6fd"}
Sep 08 20:35:18.477591 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:18.477558    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7" event={"ID":"bc17884e-e7e5-4760-8012-871ab8238e71","Type":"ContainerStarted","Data":"7866668b60137a6574840d9eda0ba6bdb98c02e80501dd267651379c09783e39"}
Sep 08 20:35:18.494359 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:18.494304    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7" podStartSLOduration=1.381092856 podStartE2EDuration="2.494285763s" totalImagesPullingTime="1.113192907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:35:18.493764906 +0000 UTC m=+571.745246262" watchObservedRunningTime="2026-09-08 20:35:18.494285763 +0000 UTC m=+571.745767168"
Sep 08 20:35:25.191986 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:25.191902    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:35:26.630741 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:26.630708    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:26.631136 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:26.630785    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:26.643278 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:26.643254    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:27.522656 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:27.522627    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7"
Sep 08 20:35:40.690024 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:40.689989    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:35:49.194602 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.194567    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"]
Sep 08 20:35:49.197820 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.197796    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:35:49.200175 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.200153    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 08 20:35:49.208433 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.208413    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"]
Sep 08 20:35:49.210659 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.210629    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21fdd723-6baa-40dd-af59-27f850ed2fc6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wqn6s\" (UID: \"21fdd723-6baa-40dd-af59-27f850ed2fc6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:35:49.210787 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.210672    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21fdd723-6baa-40dd-af59-27f850ed2fc6-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wqn6s\" (UID: \"21fdd723-6baa-40dd-af59-27f850ed2fc6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:35:49.210787 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.210728    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckk8b\" (UniqueName: \"kubernetes.io/projected/21fdd723-6baa-40dd-af59-27f850ed2fc6-kube-api-access-ckk8b\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wqn6s\" (UID: \"21fdd723-6baa-40dd-af59-27f850ed2fc6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:35:49.210787 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.210774    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21fdd723-6baa-40dd-af59-27f850ed2fc6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wqn6s\" (UID: \"21fdd723-6baa-40dd-af59-27f850ed2fc6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:35:49.210955 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.210804    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21fdd723-6baa-40dd-af59-27f850ed2fc6-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wqn6s\" (UID: \"21fdd723-6baa-40dd-af59-27f850ed2fc6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:35:49.210955 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.210834    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21fdd723-6baa-40dd-af59-27f850ed2fc6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wqn6s\" (UID: \"21fdd723-6baa-40dd-af59-27f850ed2fc6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:35:49.311591 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.311554    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21fdd723-6baa-40dd-af59-27f850ed2fc6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wqn6s\" (UID: \"21fdd723-6baa-40dd-af59-27f850ed2fc6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:35:49.311772 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.311601    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21fdd723-6baa-40dd-af59-27f850ed2fc6-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wqn6s\" (UID: \"21fdd723-6baa-40dd-af59-27f850ed2fc6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:35:49.311772 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.311667    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckk8b\" (UniqueName: \"kubernetes.io/projected/21fdd723-6baa-40dd-af59-27f850ed2fc6-kube-api-access-ckk8b\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wqn6s\" (UID: \"21fdd723-6baa-40dd-af59-27f850ed2fc6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:35:49.311895 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.311768    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21fdd723-6baa-40dd-af59-27f850ed2fc6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wqn6s\" (UID: \"21fdd723-6baa-40dd-af59-27f850ed2fc6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:35:49.311895 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.311811    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21fdd723-6baa-40dd-af59-27f850ed2fc6-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wqn6s\" (UID: \"21fdd723-6baa-40dd-af59-27f850ed2fc6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:35:49.311895 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.311844    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21fdd723-6baa-40dd-af59-27f850ed2fc6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wqn6s\" (UID: \"21fdd723-6baa-40dd-af59-27f850ed2fc6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:35:49.312047 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.311957    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21fdd723-6baa-40dd-af59-27f850ed2fc6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wqn6s\" (UID: \"21fdd723-6baa-40dd-af59-27f850ed2fc6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:35:49.312254 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.312218    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21fdd723-6baa-40dd-af59-27f850ed2fc6-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wqn6s\" (UID: \"21fdd723-6baa-40dd-af59-27f850ed2fc6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:35:49.312335 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.312309    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21fdd723-6baa-40dd-af59-27f850ed2fc6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wqn6s\" (UID: \"21fdd723-6baa-40dd-af59-27f850ed2fc6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:35:49.314203 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.314180    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21fdd723-6baa-40dd-af59-27f850ed2fc6-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wqn6s\" (UID: \"21fdd723-6baa-40dd-af59-27f850ed2fc6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:35:49.314389 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.314373    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21fdd723-6baa-40dd-af59-27f850ed2fc6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wqn6s\" (UID: \"21fdd723-6baa-40dd-af59-27f850ed2fc6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:35:49.319703 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.319680    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckk8b\" (UniqueName: \"kubernetes.io/projected/21fdd723-6baa-40dd-af59-27f850ed2fc6-kube-api-access-ckk8b\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wqn6s\" (UID: \"21fdd723-6baa-40dd-af59-27f850ed2fc6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:35:49.509199 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.509113    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:35:49.783266 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:49.783227    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"]
Sep 08 20:35:49.785592 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:35:49.785562    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21fdd723_6baa_40dd_af59_27f850ed2fc6.slice/crio-b7b1f04311001ac2760e370e933ef22724bceeae9da74e4520408e2e984071b9 WatchSource:0}: Error finding container b7b1f04311001ac2760e370e933ef22724bceeae9da74e4520408e2e984071b9: Status 404 returned error can't find the container with id b7b1f04311001ac2760e370e933ef22724bceeae9da74e4520408e2e984071b9
Sep 08 20:35:50.509695 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:50.509655    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:35:50.598330 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:50.598280    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s" event={"ID":"21fdd723-6baa-40dd-af59-27f850ed2fc6","Type":"ContainerStarted","Data":"b7b1f04311001ac2760e370e933ef22724bceeae9da74e4520408e2e984071b9"}
Sep 08 20:35:51.603203 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:51.603166    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s" event={"ID":"21fdd723-6baa-40dd-af59-27f850ed2fc6","Type":"ContainerStarted","Data":"d9fc8cb8abdd6a80237dc04066f01f0e30f30906afd7eba00bb46bc4cea3fd9d"}
Sep 08 20:35:51.622810 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:51.622753    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s" podStartSLOduration=1.7844406940000002 podStartE2EDuration="2.622734426s" totalImagesPullingTime="838.293732ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:35:51.621798068 +0000 UTC m=+604.873279425" watchObservedRunningTime="2026-09-08 20:35:51.622734426 +0000 UTC m=+604.874215784"
Sep 08 20:35:52.384746 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.384713    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"]
Sep 08 20:35:52.387973 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.387951    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:35:52.390355 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.390334    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 08 20:35:52.396610 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.396584    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"]
Sep 08 20:35:52.433201 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.433175    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68fd636d-af9b-4192-886f-bac53ce12753-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww\" (UID: \"68fd636d-af9b-4192-886f-bac53ce12753\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:35:52.433337 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.433209    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rzqn\" (UniqueName: \"kubernetes.io/projected/68fd636d-af9b-4192-886f-bac53ce12753-kube-api-access-8rzqn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww\" (UID: \"68fd636d-af9b-4192-886f-bac53ce12753\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:35:52.433337 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.433310    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68fd636d-af9b-4192-886f-bac53ce12753-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww\" (UID: \"68fd636d-af9b-4192-886f-bac53ce12753\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:35:52.433337 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.433330    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68fd636d-af9b-4192-886f-bac53ce12753-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww\" (UID: \"68fd636d-af9b-4192-886f-bac53ce12753\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:35:52.433442 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.433346    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68fd636d-af9b-4192-886f-bac53ce12753-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww\" (UID: \"68fd636d-af9b-4192-886f-bac53ce12753\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:35:52.433442 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.433360    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fd636d-af9b-4192-886f-bac53ce12753-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww\" (UID: \"68fd636d-af9b-4192-886f-bac53ce12753\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:35:52.534114 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.534077    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rzqn\" (UniqueName: \"kubernetes.io/projected/68fd636d-af9b-4192-886f-bac53ce12753-kube-api-access-8rzqn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww\" (UID: \"68fd636d-af9b-4192-886f-bac53ce12753\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:35:52.534309 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.534167    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68fd636d-af9b-4192-886f-bac53ce12753-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww\" (UID: \"68fd636d-af9b-4192-886f-bac53ce12753\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:35:52.534309 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.534193    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68fd636d-af9b-4192-886f-bac53ce12753-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww\" (UID: \"68fd636d-af9b-4192-886f-bac53ce12753\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:35:52.534309 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.534211    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68fd636d-af9b-4192-886f-bac53ce12753-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww\" (UID: \"68fd636d-af9b-4192-886f-bac53ce12753\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:35:52.534309 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.534228    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fd636d-af9b-4192-886f-bac53ce12753-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww\" (UID: \"68fd636d-af9b-4192-886f-bac53ce12753\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:35:52.534309 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.534282    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68fd636d-af9b-4192-886f-bac53ce12753-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww\" (UID: \"68fd636d-af9b-4192-886f-bac53ce12753\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:35:52.534674 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.534648    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68fd636d-af9b-4192-886f-bac53ce12753-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww\" (UID: \"68fd636d-af9b-4192-886f-bac53ce12753\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:35:52.534746 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.534719    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68fd636d-af9b-4192-886f-bac53ce12753-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww\" (UID: \"68fd636d-af9b-4192-886f-bac53ce12753\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:35:52.534785 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.534770    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fd636d-af9b-4192-886f-bac53ce12753-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww\" (UID: \"68fd636d-af9b-4192-886f-bac53ce12753\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:35:52.536521 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.536491    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68fd636d-af9b-4192-886f-bac53ce12753-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww\" (UID: \"68fd636d-af9b-4192-886f-bac53ce12753\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:35:52.536815 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.536798    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68fd636d-af9b-4192-886f-bac53ce12753-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww\" (UID: \"68fd636d-af9b-4192-886f-bac53ce12753\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:35:52.543839 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.543818    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rzqn\" (UniqueName: \"kubernetes.io/projected/68fd636d-af9b-4192-886f-bac53ce12753-kube-api-access-8rzqn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww\" (UID: \"68fd636d-af9b-4192-886f-bac53ce12753\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:35:52.697910 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.697822    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:35:52.969851 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:52.969825    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"]
Sep 08 20:35:52.972220 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:35:52.972193    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68fd636d_af9b_4192_886f_bac53ce12753.slice/crio-f1582becd483463ad026d7a48c36ed6f13f7a5b5e99ef98f5c6e35fdae8151fa WatchSource:0}: Error finding container f1582becd483463ad026d7a48c36ed6f13f7a5b5e99ef98f5c6e35fdae8151fa: Status 404 returned error can't find the container with id f1582becd483463ad026d7a48c36ed6f13f7a5b5e99ef98f5c6e35fdae8151fa
Sep 08 20:35:53.612250 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:53.612212    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww" event={"ID":"68fd636d-af9b-4192-886f-bac53ce12753","Type":"ContainerStarted","Data":"44b0a9b5ec228868461513ffa3644ae7bf75cdd3706840eb979f4b30af32a43b"}
Sep 08 20:35:53.612394 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:53.612295    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww" event={"ID":"68fd636d-af9b-4192-886f-bac53ce12753","Type":"ContainerStarted","Data":"f1582becd483463ad026d7a48c36ed6f13f7a5b5e99ef98f5c6e35fdae8151fa"}
Sep 08 20:35:53.633966 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:53.633918    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww" podStartSLOduration=1.390852664 podStartE2EDuration="1.633903955s" totalImagesPullingTime="243.051291ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:35:53.632660858 +0000 UTC m=+606.884142214" watchObservedRunningTime="2026-09-08 20:35:53.633903955 +0000 UTC m=+606.885385312"
Sep 08 20:35:57.384818 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:57.384786    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:35:59.510134 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:59.510097    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:35:59.510534 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:59.510155    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:35:59.522468 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:59.522446    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:35:59.647598 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:35:59.647572    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wqn6s"
Sep 08 20:36:02.698105 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:02.698074    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:36:02.700225 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:02.698154    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:36:02.710589 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:02.710573    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:36:03.663448 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:03.663413    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww"
Sep 08 20:36:11.193351 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.193269    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"]
Sep 08 20:36:11.196919 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.196897    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:11.199278 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.199257    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 08 20:36:11.205914 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.205893    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"]
Sep 08 20:36:11.278268 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.278221    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c1d751c-6450-4c91-8329-3cce55243335-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ktbjp\" (UID: \"9c1d751c-6450-4c91-8329-3cce55243335\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:11.278431 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.278307    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c1d751c-6450-4c91-8329-3cce55243335-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ktbjp\" (UID: \"9c1d751c-6450-4c91-8329-3cce55243335\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:11.278431 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.278339    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c1d751c-6450-4c91-8329-3cce55243335-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ktbjp\" (UID: \"9c1d751c-6450-4c91-8329-3cce55243335\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:11.278431 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.278363    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c1d751c-6450-4c91-8329-3cce55243335-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ktbjp\" (UID: \"9c1d751c-6450-4c91-8329-3cce55243335\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:11.278536 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.278427    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkk9l\" (UniqueName: \"kubernetes.io/projected/9c1d751c-6450-4c91-8329-3cce55243335-kube-api-access-jkk9l\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ktbjp\" (UID: \"9c1d751c-6450-4c91-8329-3cce55243335\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:11.278536 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.278483    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c1d751c-6450-4c91-8329-3cce55243335-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ktbjp\" (UID: \"9c1d751c-6450-4c91-8329-3cce55243335\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:11.379572 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.379535    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkk9l\" (UniqueName: \"kubernetes.io/projected/9c1d751c-6450-4c91-8329-3cce55243335-kube-api-access-jkk9l\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ktbjp\" (UID: \"9c1d751c-6450-4c91-8329-3cce55243335\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:11.379572 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.379578    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c1d751c-6450-4c91-8329-3cce55243335-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ktbjp\" (UID: \"9c1d751c-6450-4c91-8329-3cce55243335\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:11.379806 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.379605    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c1d751c-6450-4c91-8329-3cce55243335-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ktbjp\" (UID: \"9c1d751c-6450-4c91-8329-3cce55243335\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:11.379806 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.379653    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c1d751c-6450-4c91-8329-3cce55243335-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ktbjp\" (UID: \"9c1d751c-6450-4c91-8329-3cce55243335\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:11.379806 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.379668    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c1d751c-6450-4c91-8329-3cce55243335-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ktbjp\" (UID: \"9c1d751c-6450-4c91-8329-3cce55243335\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:11.379806 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.379691    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c1d751c-6450-4c91-8329-3cce55243335-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ktbjp\" (UID: \"9c1d751c-6450-4c91-8329-3cce55243335\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:11.380108 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.380081    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c1d751c-6450-4c91-8329-3cce55243335-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ktbjp\" (UID: \"9c1d751c-6450-4c91-8329-3cce55243335\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:11.380108 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.380099    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c1d751c-6450-4c91-8329-3cce55243335-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ktbjp\" (UID: \"9c1d751c-6450-4c91-8329-3cce55243335\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:11.380213 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.380154    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c1d751c-6450-4c91-8329-3cce55243335-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ktbjp\" (UID: \"9c1d751c-6450-4c91-8329-3cce55243335\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:11.381965 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.381948    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c1d751c-6450-4c91-8329-3cce55243335-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ktbjp\" (UID: \"9c1d751c-6450-4c91-8329-3cce55243335\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:11.382157 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.382142    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c1d751c-6450-4c91-8329-3cce55243335-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ktbjp\" (UID: \"9c1d751c-6450-4c91-8329-3cce55243335\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:11.387324 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.387296    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkk9l\" (UniqueName: \"kubernetes.io/projected/9c1d751c-6450-4c91-8329-3cce55243335-kube-api-access-jkk9l\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ktbjp\" (UID: \"9c1d751c-6450-4c91-8329-3cce55243335\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:11.508497 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.508414    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:11.777702 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.777676    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"]
Sep 08 20:36:11.779911 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:36:11.779880    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1d751c_6450_4c91_8329_3cce55243335.slice/crio-d80418138f728d9f1a4f276a65627cbc5431b531dd5d3294f27d32d8052783dd WatchSource:0}: Error finding container d80418138f728d9f1a4f276a65627cbc5431b531dd5d3294f27d32d8052783dd: Status 404 returned error can't find the container with id d80418138f728d9f1a4f276a65627cbc5431b531dd5d3294f27d32d8052783dd
Sep 08 20:36:11.781594 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:11.781578    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 20:36:12.689272 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:12.689223    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp" event={"ID":"9c1d751c-6450-4c91-8329-3cce55243335","Type":"ContainerStarted","Data":"e233b04bf7525d4ac3c1124118a6e6f17bc0b36b5360e3113fe3916c23910ff4"}
Sep 08 20:36:12.689651 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:12.689300    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp" event={"ID":"9c1d751c-6450-4c91-8329-3cce55243335","Type":"ContainerStarted","Data":"d80418138f728d9f1a4f276a65627cbc5431b531dd5d3294f27d32d8052783dd"}
Sep 08 20:36:12.707459 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:12.707402    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp" podStartSLOduration=1.524182545 podStartE2EDuration="1.707379782s" totalImagesPullingTime="183.197237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:36:12.705192625 +0000 UTC m=+625.956673985" watchObservedRunningTime="2026-09-08 20:36:12.707379782 +0000 UTC m=+625.958861142"
Sep 08 20:36:13.094750 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:13.094714    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:36:21.509534 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:21.509499    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:21.510102 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:21.509567    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:21.522074 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:21.522044    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:21.737601 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:21.737572    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ktbjp"
Sep 08 20:36:25.196851 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:36:25.196769    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:38:21.297849 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:21.297815    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:38:30.956882 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:30.956849    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh"]
Sep 08 20:38:30.960249 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:30.960219    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh"
Sep 08 20:38:30.962622 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:30.962586    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c73d714b-dockercfg-m27kp\""
Sep 08 20:38:30.962742 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:30.962730    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c73d714b\""
Sep 08 20:38:30.979265 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:30.979214    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh"]
Sep 08 20:38:30.982994 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:30.982964    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d57595ec-4b05-4e3a-a231-b9014c020d00-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh\" (UID: \"d57595ec-4b05-4e3a-a231-b9014c020d00\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh"
Sep 08 20:38:30.983082 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:30.983014    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgs59\" (UniqueName: \"kubernetes.io/projected/d57595ec-4b05-4e3a-a231-b9014c020d00-kube-api-access-bgs59\") pod \"payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh\" (UID: \"d57595ec-4b05-4e3a-a231-b9014c020d00\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh"
Sep 08 20:38:31.083599 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:31.083564    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d57595ec-4b05-4e3a-a231-b9014c020d00-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh\" (UID: \"d57595ec-4b05-4e3a-a231-b9014c020d00\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh"
Sep 08 20:38:31.083788 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:31.083610    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgs59\" (UniqueName: \"kubernetes.io/projected/d57595ec-4b05-4e3a-a231-b9014c020d00-kube-api-access-bgs59\") pod \"payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh\" (UID: \"d57595ec-4b05-4e3a-a231-b9014c020d00\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh"
Sep 08 20:38:31.084353 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:31.084328    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d57595ec-4b05-4e3a-a231-b9014c020d00-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh\" (UID: \"d57595ec-4b05-4e3a-a231-b9014c020d00\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh"
Sep 08 20:38:31.092005 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:31.091982    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgs59\" (UniqueName: \"kubernetes.io/projected/d57595ec-4b05-4e3a-a231-b9014c020d00-kube-api-access-bgs59\") pod \"payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh\" (UID: \"d57595ec-4b05-4e3a-a231-b9014c020d00\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh"
Sep 08 20:38:31.270754 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:31.270673    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh"
Sep 08 20:38:31.538227 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:31.538154    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh"]
Sep 08 20:38:31.541505 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:38:31.541477    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd57595ec_4b05_4e3a_a231_b9014c020d00.slice/crio-e542b6d3d91febcb9e88c02b93de02dd4234f673a1a81dbb31fa5d6b09a610cc WatchSource:0}: Error finding container e542b6d3d91febcb9e88c02b93de02dd4234f673a1a81dbb31fa5d6b09a610cc: Status 404 returned error can't find the container with id e542b6d3d91febcb9e88c02b93de02dd4234f673a1a81dbb31fa5d6b09a610cc
Sep 08 20:38:32.256320 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:32.256275    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh" event={"ID":"d57595ec-4b05-4e3a-a231-b9014c020d00","Type":"ContainerStarted","Data":"e542b6d3d91febcb9e88c02b93de02dd4234f673a1a81dbb31fa5d6b09a610cc"}
Sep 08 20:38:34.267543 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:34.267510    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh" event={"ID":"d57595ec-4b05-4e3a-a231-b9014c020d00","Type":"ContainerStarted","Data":"9e8e786a0d45cef5968342b09525120c3c8a4c672c1ca4fbb02a8ffe0e8f2a64"}
Sep 08 20:38:34.267933 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:34.267605    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh"
Sep 08 20:38:34.283068 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:34.283008    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh" podStartSLOduration=1.71683746 podStartE2EDuration="4.28298921s" totalImagesPullingTime="2.56615175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:38:34.282293479 +0000 UTC m=+767.533774836" watchObservedRunningTime="2026-09-08 20:38:34.28298921 +0000 UTC m=+767.534470568"
Sep 08 20:38:37.596655 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:37.596617    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:38:45.273434 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:45.273402    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh"
Sep 08 20:38:51.300672 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.300630    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2"]
Sep 08 20:38:51.304030 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.304008    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2"
Sep 08 20:38:51.306251 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.306213    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a6137b-dockercfg-cznfs\""
Sep 08 20:38:51.306251 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.306225    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a6137b\""
Sep 08 20:38:51.314819 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.314797    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2"]
Sep 08 20:38:51.322188 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.322169    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx"]
Sep 08 20:38:51.325408 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.325384    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx"
Sep 08 20:38:51.335750 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.335730    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx"]
Sep 08 20:38:51.363706 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.363669    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824b6925-412a-4bd6-b80a-288a8c27716d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2\" (UID: \"824b6925-412a-4bd6-b80a-288a8c27716d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2"
Sep 08 20:38:51.363841 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.363737    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzd2k\" (UniqueName: \"kubernetes.io/projected/824b6925-412a-4bd6-b80a-288a8c27716d-kube-api-access-gzd2k\") pod \"payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2\" (UID: \"824b6925-412a-4bd6-b80a-288a8c27716d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2"
Sep 08 20:38:51.464736 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.464697    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgqb4\" (UniqueName: \"kubernetes.io/projected/04f7f517-cbc8-4e81-9155-a78ceec8fb06-kube-api-access-tgqb4\") pod \"payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx\" (UID: \"04f7f517-cbc8-4e81-9155-a78ceec8fb06\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx"
Sep 08 20:38:51.464902 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.464750    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824b6925-412a-4bd6-b80a-288a8c27716d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2\" (UID: \"824b6925-412a-4bd6-b80a-288a8c27716d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2"
Sep 08 20:38:51.464902 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.464816    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzd2k\" (UniqueName: \"kubernetes.io/projected/824b6925-412a-4bd6-b80a-288a8c27716d-kube-api-access-gzd2k\") pod \"payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2\" (UID: \"824b6925-412a-4bd6-b80a-288a8c27716d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2"
Sep 08 20:38:51.465036 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.464908    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04f7f517-cbc8-4e81-9155-a78ceec8fb06-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx\" (UID: \"04f7f517-cbc8-4e81-9155-a78ceec8fb06\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx"
Sep 08 20:38:51.465539 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.465521    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824b6925-412a-4bd6-b80a-288a8c27716d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2\" (UID: \"824b6925-412a-4bd6-b80a-288a8c27716d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2"
Sep 08 20:38:51.472584 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.472556    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzd2k\" (UniqueName: \"kubernetes.io/projected/824b6925-412a-4bd6-b80a-288a8c27716d-kube-api-access-gzd2k\") pod \"payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2\" (UID: \"824b6925-412a-4bd6-b80a-288a8c27716d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2"
Sep 08 20:38:51.565457 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.565352    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04f7f517-cbc8-4e81-9155-a78ceec8fb06-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx\" (UID: \"04f7f517-cbc8-4e81-9155-a78ceec8fb06\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx"
Sep 08 20:38:51.565457 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.565407    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgqb4\" (UniqueName: \"kubernetes.io/projected/04f7f517-cbc8-4e81-9155-a78ceec8fb06-kube-api-access-tgqb4\") pod \"payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx\" (UID: \"04f7f517-cbc8-4e81-9155-a78ceec8fb06\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx"
Sep 08 20:38:51.566047 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.566025    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04f7f517-cbc8-4e81-9155-a78ceec8fb06-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx\" (UID: \"04f7f517-cbc8-4e81-9155-a78ceec8fb06\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx"
Sep 08 20:38:51.573786 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.573755    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgqb4\" (UniqueName: \"kubernetes.io/projected/04f7f517-cbc8-4e81-9155-a78ceec8fb06-kube-api-access-tgqb4\") pod \"payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx\" (UID: \"04f7f517-cbc8-4e81-9155-a78ceec8fb06\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx"
Sep 08 20:38:51.615205 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.615174    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2"
Sep 08 20:38:51.636476 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.636445    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx"
Sep 08 20:38:51.894435 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.894405    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2"]
Sep 08 20:38:51.894952 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:38:51.894922    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824b6925_412a_4bd6_b80a_288a8c27716d.slice/crio-0c3f152d0e4feb37774637002cbd1346d9bee9ead406fff2c7dd22b3f41e98eb WatchSource:0}: Error finding container 0c3f152d0e4feb37774637002cbd1346d9bee9ead406fff2c7dd22b3f41e98eb: Status 404 returned error can't find the container with id 0c3f152d0e4feb37774637002cbd1346d9bee9ead406fff2c7dd22b3f41e98eb
Sep 08 20:38:51.918650 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:51.918625    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx"]
Sep 08 20:38:51.922183 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:38:51.922161    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f7f517_cbc8_4e81_9155_a78ceec8fb06.slice/crio-43378e43b5c7d9d4c7d37b11c42aeb8a0b11786b3f549f84e355dd4e9d63512a WatchSource:0}: Error finding container 43378e43b5c7d9d4c7d37b11c42aeb8a0b11786b3f549f84e355dd4e9d63512a: Status 404 returned error can't find the container with id 43378e43b5c7d9d4c7d37b11c42aeb8a0b11786b3f549f84e355dd4e9d63512a
Sep 08 20:38:52.348608 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:52.348572    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx" event={"ID":"04f7f517-cbc8-4e81-9155-a78ceec8fb06","Type":"ContainerStarted","Data":"ffbb056666d91baacfb2768b7bb9252106e64f7c9cf411ab19c784ebc7e00d7f"}
Sep 08 20:38:52.349160 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:52.348637    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx" event={"ID":"04f7f517-cbc8-4e81-9155-a78ceec8fb06","Type":"ContainerStarted","Data":"43378e43b5c7d9d4c7d37b11c42aeb8a0b11786b3f549f84e355dd4e9d63512a"}
Sep 08 20:38:52.349160 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:52.348666    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx"
Sep 08 20:38:52.350203 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:52.350174    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2" event={"ID":"824b6925-412a-4bd6-b80a-288a8c27716d","Type":"ContainerStarted","Data":"9c335d5b74aee4c38729e9bdd5d489adcea73677f116af5cbc1dbda16a32335d"}
Sep 08 20:38:52.350363 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:52.350228    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2"
Sep 08 20:38:52.350363 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:52.350278    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2" event={"ID":"824b6925-412a-4bd6-b80a-288a8c27716d","Type":"ContainerStarted","Data":"0c3f152d0e4feb37774637002cbd1346d9bee9ead406fff2c7dd22b3f41e98eb"}
Sep 08 20:38:52.365281 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:52.365220    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx" podStartSLOduration=1.365205618 podStartE2EDuration="1.365205618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:38:52.36366017 +0000 UTC m=+785.615141527" watchObservedRunningTime="2026-09-08 20:38:52.365205618 +0000 UTC m=+785.616687017"
Sep 08 20:38:52.382109 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:52.382054    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2" podStartSLOduration=1.382035092 podStartE2EDuration="1.382035092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:38:52.380076719 +0000 UTC m=+785.631558069" watchObservedRunningTime="2026-09-08 20:38:52.382035092 +0000 UTC m=+785.633516452"
Sep 08 20:38:54.233470 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:54.233436    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c73d714b-tbnss"]
Sep 08 20:38:54.236932 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:54.236908    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c73d714b-tbnss"
Sep 08 20:38:54.239003 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:54.238981    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 20:38:54.239109 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:54.239031    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2b2qs\""
Sep 08 20:38:54.243290 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:54.243271    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c73d714b-tbnss"]
Sep 08 20:38:54.392128 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:54.392098    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e573006-e6ee-46c4-a772-7c4898869fd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c73d714b-tbnss\" (UID: \"1e573006-e6ee-46c4-a772-7c4898869fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c73d714b-tbnss"
Sep 08 20:38:54.493536 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:54.493433    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e573006-e6ee-46c4-a772-7c4898869fd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c73d714b-tbnss\" (UID: \"1e573006-e6ee-46c4-a772-7c4898869fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c73d714b-tbnss"
Sep 08 20:38:54.494071 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:54.494047    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e573006-e6ee-46c4-a772-7c4898869fd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c73d714b-tbnss\" (UID: \"1e573006-e6ee-46c4-a772-7c4898869fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c73d714b-tbnss"
Sep 08 20:38:54.548693 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:54.548651    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c73d714b-tbnss"
Sep 08 20:38:54.817785 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:54.817632    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c73d714b-tbnss"]
Sep 08 20:38:54.820123 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:38:54.820092    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e573006_e6ee_46c4_a772_7c4898869fd7.slice/crio-d45090e6317691a967b13e6b4734e957e7956940985731dcf0d03dc09dce0e9f WatchSource:0}: Error finding container d45090e6317691a967b13e6b4734e957e7956940985731dcf0d03dc09dce0e9f: Status 404 returned error can't find the container with id d45090e6317691a967b13e6b4734e957e7956940985731dcf0d03dc09dce0e9f
Sep 08 20:38:55.364000 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:55.363969    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c73d714b-tbnss" event={"ID":"1e573006-e6ee-46c4-a772-7c4898869fd7","Type":"ContainerStarted","Data":"d45090e6317691a967b13e6b4734e957e7956940985731dcf0d03dc09dce0e9f"}
Sep 08 20:38:57.373076 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:57.373042    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c73d714b-tbnss" event={"ID":"1e573006-e6ee-46c4-a772-7c4898869fd7","Type":"ContainerStarted","Data":"a7505290047216be9e573d4e65bcdf58100a1bbaed6ac20e6395871c03752e2e"}
Sep 08 20:38:57.388375 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:57.388327    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c73d714b-tbnss" podStartSLOduration=1.305099724 podStartE2EDuration="3.388311635s" totalImagesPullingTime="2.083211911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:38:57.386805442 +0000 UTC m=+790.638286799" watchObservedRunningTime="2026-09-08 20:38:57.388311635 +0000 UTC m=+790.639792991"
Sep 08 20:38:57.692395 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:57.692303    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:38:58.378339 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:58.378306    2552 generic.go:356] "Generic (PLEG): container finished" podID="1e573006-e6ee-46c4-a772-7c4898869fd7" containerID="a7505290047216be9e573d4e65bcdf58100a1bbaed6ac20e6395871c03752e2e" exitCode=0
Sep 08 20:38:58.378778 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:58.378369    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c73d714b-tbnss" event={"ID":"1e573006-e6ee-46c4-a772-7c4898869fd7","Type":"ContainerDied","Data":"a7505290047216be9e573d4e65bcdf58100a1bbaed6ac20e6395871c03752e2e"}
Sep 08 20:38:59.521169 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:59.521146    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c73d714b-tbnss"
Sep 08 20:38:59.642599 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:59.642517    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e573006-e6ee-46c4-a772-7c4898869fd7-maas-api-service-ca\") pod \"1e573006-e6ee-46c4-a772-7c4898869fd7\" (UID: \"1e573006-e6ee-46c4-a772-7c4898869fd7\") "
Sep 08 20:38:59.642956 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:59.642924    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e573006-e6ee-46c4-a772-7c4898869fd7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1e573006-e6ee-46c4-a772-7c4898869fd7" (UID: "1e573006-e6ee-46c4-a772-7c4898869fd7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:38:59.743509 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:38:59.743477    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e573006-e6ee-46c4-a772-7c4898869fd7-maas-api-service-ca\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:39:00.388411 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:00.388382    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c73d714b-tbnss"
Sep 08 20:39:00.388576 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:00.388382    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c73d714b-tbnss" event={"ID":"1e573006-e6ee-46c4-a772-7c4898869fd7","Type":"ContainerDied","Data":"d45090e6317691a967b13e6b4734e957e7956940985731dcf0d03dc09dce0e9f"}
Sep 08 20:39:00.388576 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:00.388512    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d45090e6317691a967b13e6b4734e957e7956940985731dcf0d03dc09dce0e9f"
Sep 08 20:39:03.356426 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:03.356384    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2"
Sep 08 20:39:03.356878 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:03.356468    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx"
Sep 08 20:39:10.835560 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:10.835471    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh"]
Sep 08 20:39:10.835975 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:10.835698    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh" podUID="d57595ec-4b05-4e3a-a231-b9014c020d00" containerName="payload-pre-processing" containerID="cri-o://9e8e786a0d45cef5968342b09525120c3c8a4c672c1ca4fbb02a8ffe0e8f2a64" gracePeriod=30
Sep 08 20:39:11.000233 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:11.000199    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c73d714b-tbnss"]
Sep 08 20:39:11.004867 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:11.004842    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c73d714b-tbnss"]
Sep 08 20:39:11.102337 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:11.102310    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh"
Sep 08 20:39:11.143046 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:11.143016    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d57595ec-4b05-4e3a-a231-b9014c020d00-plugins-config-volume\") pod \"d57595ec-4b05-4e3a-a231-b9014c020d00\" (UID: \"d57595ec-4b05-4e3a-a231-b9014c020d00\") "
Sep 08 20:39:11.143226 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:11.143125    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgs59\" (UniqueName: \"kubernetes.io/projected/d57595ec-4b05-4e3a-a231-b9014c020d00-kube-api-access-bgs59\") pod \"d57595ec-4b05-4e3a-a231-b9014c020d00\" (UID: \"d57595ec-4b05-4e3a-a231-b9014c020d00\") "
Sep 08 20:39:11.143494 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:11.143467    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d57595ec-4b05-4e3a-a231-b9014c020d00-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d57595ec-4b05-4e3a-a231-b9014c020d00" (UID: "d57595ec-4b05-4e3a-a231-b9014c020d00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:39:11.145334 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:11.145308    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d57595ec-4b05-4e3a-a231-b9014c020d00-kube-api-access-bgs59" (OuterVolumeSpecName: "kube-api-access-bgs59") pod "d57595ec-4b05-4e3a-a231-b9014c020d00" (UID: "d57595ec-4b05-4e3a-a231-b9014c020d00"). InnerVolumeSpecName "kube-api-access-bgs59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:39:11.224507 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:11.224477    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e573006-e6ee-46c4-a772-7c4898869fd7" path="/var/lib/kubelet/pods/1e573006-e6ee-46c4-a772-7c4898869fd7/volumes"
Sep 08 20:39:11.243665 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:11.243643    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgs59\" (UniqueName: \"kubernetes.io/projected/d57595ec-4b05-4e3a-a231-b9014c020d00-kube-api-access-bgs59\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:39:11.243665 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:11.243664    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d57595ec-4b05-4e3a-a231-b9014c020d00-plugins-config-volume\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:39:11.438919 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:11.438835    2552 generic.go:356] "Generic (PLEG): container finished" podID="d57595ec-4b05-4e3a-a231-b9014c020d00" containerID="9e8e786a0d45cef5968342b09525120c3c8a4c672c1ca4fbb02a8ffe0e8f2a64" exitCode=0
Sep 08 20:39:11.438919 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:11.438877    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh"
Sep 08 20:39:11.439086 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:11.438917    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh" event={"ID":"d57595ec-4b05-4e3a-a231-b9014c020d00","Type":"ContainerDied","Data":"9e8e786a0d45cef5968342b09525120c3c8a4c672c1ca4fbb02a8ffe0e8f2a64"}
Sep 08 20:39:11.439086 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:11.438969    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh" event={"ID":"d57595ec-4b05-4e3a-a231-b9014c020d00","Type":"ContainerDied","Data":"e542b6d3d91febcb9e88c02b93de02dd4234f673a1a81dbb31fa5d6b09a610cc"}
Sep 08 20:39:11.439656 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:11.439637    2552 scope.go:117] "RemoveContainer" containerID="9e8e786a0d45cef5968342b09525120c3c8a4c672c1ca4fbb02a8ffe0e8f2a64"
Sep 08 20:39:11.448165 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:11.448145    2552 scope.go:117] "RemoveContainer" containerID="9e8e786a0d45cef5968342b09525120c3c8a4c672c1ca4fbb02a8ffe0e8f2a64"
Sep 08 20:39:11.448422 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:39:11.448397    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e8e786a0d45cef5968342b09525120c3c8a4c672c1ca4fbb02a8ffe0e8f2a64\": container with ID starting with 9e8e786a0d45cef5968342b09525120c3c8a4c672c1ca4fbb02a8ffe0e8f2a64 not found: ID does not exist" containerID="9e8e786a0d45cef5968342b09525120c3c8a4c672c1ca4fbb02a8ffe0e8f2a64"
Sep 08 20:39:11.448463 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:11.448432    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8e786a0d45cef5968342b09525120c3c8a4c672c1ca4fbb02a8ffe0e8f2a64"} err="failed to get container status \"9e8e786a0d45cef5968342b09525120c3c8a4c672c1ca4fbb02a8ffe0e8f2a64\": rpc error: code = NotFound desc = could not find container \"9e8e786a0d45cef5968342b09525120c3c8a4c672c1ca4fbb02a8ffe0e8f2a64\": container with ID starting with 9e8e786a0d45cef5968342b09525120c3c8a4c672c1ca4fbb02a8ffe0e8f2a64 not found: ID does not exist"
Sep 08 20:39:11.454762 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:11.454743    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh"]
Sep 08 20:39:11.457175 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:11.457153    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c73d714b-86d464646-gxmjh"]
Sep 08 20:39:13.224364 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:13.224328    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d57595ec-4b05-4e3a-a231-b9014c020d00" path="/var/lib/kubelet/pods/d57595ec-4b05-4e3a-a231-b9014c020d00/volumes"
Sep 08 20:39:15.537833 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.537798    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j"]
Sep 08 20:39:15.538212 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.538103    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e573006-e6ee-46c4-a772-7c4898869fd7" containerName="revoke-keys"
Sep 08 20:39:15.538212 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.538115    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e573006-e6ee-46c4-a772-7c4898869fd7" containerName="revoke-keys"
Sep 08 20:39:15.538212 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.538132    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d57595ec-4b05-4e3a-a231-b9014c020d00" containerName="payload-pre-processing"
Sep 08 20:39:15.538212 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.538137    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d57595ec-4b05-4e3a-a231-b9014c020d00" containerName="payload-pre-processing"
Sep 08 20:39:15.538212 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.538205    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e573006-e6ee-46c4-a772-7c4898869fd7" containerName="revoke-keys"
Sep 08 20:39:15.538212 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.538214    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d57595ec-4b05-4e3a-a231-b9014c020d00" containerName="payload-pre-processing"
Sep 08 20:39:15.541065 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.541050    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j"
Sep 08 20:39:15.543574 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.543542    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-4d417ac0\""
Sep 08 20:39:15.543694 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.543675    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-4d417ac0-dockercfg-dgdtm\""
Sep 08 20:39:15.561400 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.561375    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j"]
Sep 08 20:39:15.582036 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.582005    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a86485db-175f-46df-a1ce-e2957d064a3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j\" (UID: \"a86485db-175f-46df-a1ce-e2957d064a3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j"
Sep 08 20:39:15.582177 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.582052    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj8bx\" (UniqueName: \"kubernetes.io/projected/a86485db-175f-46df-a1ce-e2957d064a3c-kube-api-access-dj8bx\") pod \"payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j\" (UID: \"a86485db-175f-46df-a1ce-e2957d064a3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j"
Sep 08 20:39:15.622371 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.622334    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7"]
Sep 08 20:39:15.625565 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.625547    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7"
Sep 08 20:39:15.639853 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.639829    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7"]
Sep 08 20:39:15.683480 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.683445    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a86485db-175f-46df-a1ce-e2957d064a3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j\" (UID: \"a86485db-175f-46df-a1ce-e2957d064a3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j"
Sep 08 20:39:15.683480 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.683494    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj8bx\" (UniqueName: \"kubernetes.io/projected/a86485db-175f-46df-a1ce-e2957d064a3c-kube-api-access-dj8bx\") pod \"payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j\" (UID: \"a86485db-175f-46df-a1ce-e2957d064a3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j"
Sep 08 20:39:15.683688 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.683521    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mp6z\" (UniqueName: \"kubernetes.io/projected/74bcca5d-b5bf-4bcd-ac0b-8805bcb15906-kube-api-access-9mp6z\") pod \"payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7\" (UID: \"74bcca5d-b5bf-4bcd-ac0b-8805bcb15906\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7"
Sep 08 20:39:15.683688 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.683555    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bcca5d-b5bf-4bcd-ac0b-8805bcb15906-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7\" (UID: \"74bcca5d-b5bf-4bcd-ac0b-8805bcb15906\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7"
Sep 08 20:39:15.684225 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.684204    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a86485db-175f-46df-a1ce-e2957d064a3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j\" (UID: \"a86485db-175f-46df-a1ce-e2957d064a3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j"
Sep 08 20:39:15.692178 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.692150    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj8bx\" (UniqueName: \"kubernetes.io/projected/a86485db-175f-46df-a1ce-e2957d064a3c-kube-api-access-dj8bx\") pod \"payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j\" (UID: \"a86485db-175f-46df-a1ce-e2957d064a3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j"
Sep 08 20:39:15.785005 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.784969    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mp6z\" (UniqueName: \"kubernetes.io/projected/74bcca5d-b5bf-4bcd-ac0b-8805bcb15906-kube-api-access-9mp6z\") pod \"payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7\" (UID: \"74bcca5d-b5bf-4bcd-ac0b-8805bcb15906\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7"
Sep 08 20:39:15.785005 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.785010    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bcca5d-b5bf-4bcd-ac0b-8805bcb15906-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7\" (UID: \"74bcca5d-b5bf-4bcd-ac0b-8805bcb15906\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7"
Sep 08 20:39:15.785720 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.785699    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bcca5d-b5bf-4bcd-ac0b-8805bcb15906-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7\" (UID: \"74bcca5d-b5bf-4bcd-ac0b-8805bcb15906\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7"
Sep 08 20:39:15.793265 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.793184    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mp6z\" (UniqueName: \"kubernetes.io/projected/74bcca5d-b5bf-4bcd-ac0b-8805bcb15906-kube-api-access-9mp6z\") pod \"payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7\" (UID: \"74bcca5d-b5bf-4bcd-ac0b-8805bcb15906\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7"
Sep 08 20:39:15.851790 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.851765    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j"
Sep 08 20:39:15.935052 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:15.935027    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7"
Sep 08 20:39:16.121047 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:16.121017    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j"]
Sep 08 20:39:16.123123 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:39:16.123097    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda86485db_175f_46df_a1ce_e2957d064a3c.slice/crio-35b676cb248e29e5a47692b0c62adfc41e35e0d48c5c46f909a0a619281638e2 WatchSource:0}: Error finding container 35b676cb248e29e5a47692b0c62adfc41e35e0d48c5c46f909a0a619281638e2: Status 404 returned error can't find the container with id 35b676cb248e29e5a47692b0c62adfc41e35e0d48c5c46f909a0a619281638e2
Sep 08 20:39:16.212343 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:16.212320    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7"]
Sep 08 20:39:16.214686 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:39:16.214657    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74bcca5d_b5bf_4bcd_ac0b_8805bcb15906.slice/crio-09e8dcc2f0a3352b3a3a32526504e4e5ce4c6c639af021cb6b46ba2f1b192a1e WatchSource:0}: Error finding container 09e8dcc2f0a3352b3a3a32526504e4e5ce4c6c639af021cb6b46ba2f1b192a1e: Status 404 returned error can't find the container with id 09e8dcc2f0a3352b3a3a32526504e4e5ce4c6c639af021cb6b46ba2f1b192a1e
Sep 08 20:39:16.294054 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:16.294022    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:39:16.464871 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:16.464773    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j" event={"ID":"a86485db-175f-46df-a1ce-e2957d064a3c","Type":"ContainerStarted","Data":"4347e8f3e17c01369b6d7114d9e538615aa2f600d21d2f5b06acabe3dd56be04"}
Sep 08 20:39:16.464871 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:16.464847    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j"
Sep 08 20:39:16.464871 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:16.464873    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j" event={"ID":"a86485db-175f-46df-a1ce-e2957d064a3c","Type":"ContainerStarted","Data":"35b676cb248e29e5a47692b0c62adfc41e35e0d48c5c46f909a0a619281638e2"}
Sep 08 20:39:16.466383 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:16.466357    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7" event={"ID":"74bcca5d-b5bf-4bcd-ac0b-8805bcb15906","Type":"ContainerStarted","Data":"450a6b379c7a95f2e29e6612ad660b55601c98c05ba599423fe4586397c05830"}
Sep 08 20:39:16.466480 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:16.466399    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7" event={"ID":"74bcca5d-b5bf-4bcd-ac0b-8805bcb15906","Type":"ContainerStarted","Data":"09e8dcc2f0a3352b3a3a32526504e4e5ce4c6c639af021cb6b46ba2f1b192a1e"}
Sep 08 20:39:16.466548 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:16.466497    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7"
Sep 08 20:39:16.479927 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:16.479865    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j" podStartSLOduration=1.47985102 podStartE2EDuration="1.47985102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:39:16.478715332 +0000 UTC m=+809.730196686" watchObservedRunningTime="2026-09-08 20:39:16.47985102 +0000 UTC m=+809.731332377"
Sep 08 20:39:16.495412 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:16.495374    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7" podStartSLOduration=1.495360897 podStartE2EDuration="1.495360897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:39:16.493912992 +0000 UTC m=+809.745394351" watchObservedRunningTime="2026-09-08 20:39:16.495360897 +0000 UTC m=+809.746842254"
Sep 08 20:39:27.473143 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:27.473112    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j"
Sep 08 20:39:27.473598 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:27.473180    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7"
Sep 08 20:39:38.198019 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:38.197986    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:39:39.616860 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:39.616828    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d417ac0-cwngx"]
Sep 08 20:39:39.624758 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:39.624729    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d417ac0-cwngx"
Sep 08 20:39:39.626865 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:39.626838    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d417ac0-cwngx"]
Sep 08 20:39:39.627340 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:39.627318    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 20:39:39.627578 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:39.627344    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2b2qs\""
Sep 08 20:39:39.696315 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:39.696280    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee17c3b-ad74-49fa-9a0d-78a69bb7b656-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d417ac0-cwngx\" (UID: \"3ee17c3b-ad74-49fa-9a0d-78a69bb7b656\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d417ac0-cwngx"
Sep 08 20:39:39.797336 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:39.797300    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee17c3b-ad74-49fa-9a0d-78a69bb7b656-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d417ac0-cwngx\" (UID: \"3ee17c3b-ad74-49fa-9a0d-78a69bb7b656\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d417ac0-cwngx"
Sep 08 20:39:39.797951 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:39.797929    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee17c3b-ad74-49fa-9a0d-78a69bb7b656-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d417ac0-cwngx\" (UID: \"3ee17c3b-ad74-49fa-9a0d-78a69bb7b656\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d417ac0-cwngx"
Sep 08 20:39:39.936650 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:39.936561    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d417ac0-cwngx"
Sep 08 20:39:40.206293 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:40.206180    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d417ac0-cwngx"]
Sep 08 20:39:40.209281 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:39:40.209247    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee17c3b_ad74_49fa_9a0d_78a69bb7b656.slice/crio-17d6763499398c50bb227f9d990cb5497eb5b330764e9fa88bb7ac44b89581f9 WatchSource:0}: Error finding container 17d6763499398c50bb227f9d990cb5497eb5b330764e9fa88bb7ac44b89581f9: Status 404 returned error can't find the container with id 17d6763499398c50bb227f9d990cb5497eb5b330764e9fa88bb7ac44b89581f9
Sep 08 20:39:40.573921 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:40.573833    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d417ac0-cwngx" event={"ID":"3ee17c3b-ad74-49fa-9a0d-78a69bb7b656","Type":"ContainerStarted","Data":"62aa163d8885ad976c62aab88b64ad0e3dcafa8b9d0f6fdc25acc6b563407e19"}
Sep 08 20:39:40.573921 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:40.573887    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d417ac0-cwngx" event={"ID":"3ee17c3b-ad74-49fa-9a0d-78a69bb7b656","Type":"ContainerStarted","Data":"17d6763499398c50bb227f9d990cb5497eb5b330764e9fa88bb7ac44b89581f9"}
Sep 08 20:39:40.588270 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:40.588208    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d417ac0-cwngx" podStartSLOduration=1.5881919610000002 podStartE2EDuration="1.588191961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:39:40.586714049 +0000 UTC m=+833.838195406" watchObservedRunningTime="2026-09-08 20:39:40.588191961 +0000 UTC m=+833.839673318"
Sep 08 20:39:41.579749 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:41.579707    2552 generic.go:356] "Generic (PLEG): container finished" podID="3ee17c3b-ad74-49fa-9a0d-78a69bb7b656" containerID="62aa163d8885ad976c62aab88b64ad0e3dcafa8b9d0f6fdc25acc6b563407e19" exitCode=0
Sep 08 20:39:41.580187 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:41.579772    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d417ac0-cwngx" event={"ID":"3ee17c3b-ad74-49fa-9a0d-78a69bb7b656","Type":"ContainerDied","Data":"62aa163d8885ad976c62aab88b64ad0e3dcafa8b9d0f6fdc25acc6b563407e19"}
Sep 08 20:39:42.714176 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:42.714153    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d417ac0-cwngx"
Sep 08 20:39:42.822854 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:42.822816    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee17c3b-ad74-49fa-9a0d-78a69bb7b656-maas-api-service-ca\") pod \"3ee17c3b-ad74-49fa-9a0d-78a69bb7b656\" (UID: \"3ee17c3b-ad74-49fa-9a0d-78a69bb7b656\") "
Sep 08 20:39:42.823202 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:42.823174    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ee17c3b-ad74-49fa-9a0d-78a69bb7b656-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3ee17c3b-ad74-49fa-9a0d-78a69bb7b656" (UID: "3ee17c3b-ad74-49fa-9a0d-78a69bb7b656"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:39:42.924172 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:42.924080    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee17c3b-ad74-49fa-9a0d-78a69bb7b656-maas-api-service-ca\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:39:43.590325 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:43.590294    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d417ac0-cwngx"
Sep 08 20:39:43.590515 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:43.590290    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d417ac0-cwngx" event={"ID":"3ee17c3b-ad74-49fa-9a0d-78a69bb7b656","Type":"ContainerDied","Data":"17d6763499398c50bb227f9d990cb5497eb5b330764e9fa88bb7ac44b89581f9"}
Sep 08 20:39:43.590515 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:43.590419    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17d6763499398c50bb227f9d990cb5497eb5b330764e9fa88bb7ac44b89581f9"
Sep 08 20:39:46.276751 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.276716    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7"]
Sep 08 20:39:46.277506 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.277470    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7" podUID="74bcca5d-b5bf-4bcd-ac0b-8805bcb15906" containerName="payload-processing" containerID="cri-o://450a6b379c7a95f2e29e6612ad660b55601c98c05ba599423fe4586397c05830" gracePeriod=30
Sep 08 20:39:46.282232 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.282207    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j"]
Sep 08 20:39:46.282662 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.282610    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j" podUID="a86485db-175f-46df-a1ce-e2957d064a3c" containerName="payload-pre-processing" containerID="cri-o://4347e8f3e17c01369b6d7114d9e538615aa2f600d21d2f5b06acabe3dd56be04" gracePeriod=30
Sep 08 20:39:46.437468 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.437435    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d417ac0-cwngx"]
Sep 08 20:39:46.440777 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.440752    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d417ac0-cwngx"]
Sep 08 20:39:46.559062 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.559040    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7"
Sep 08 20:39:46.561497 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.561473    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j"
Sep 08 20:39:46.604683 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.604617    2552 generic.go:356] "Generic (PLEG): container finished" podID="a86485db-175f-46df-a1ce-e2957d064a3c" containerID="4347e8f3e17c01369b6d7114d9e538615aa2f600d21d2f5b06acabe3dd56be04" exitCode=0
Sep 08 20:39:46.604683 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.604660    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j"
Sep 08 20:39:46.604906 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.604737    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j" event={"ID":"a86485db-175f-46df-a1ce-e2957d064a3c","Type":"ContainerDied","Data":"4347e8f3e17c01369b6d7114d9e538615aa2f600d21d2f5b06acabe3dd56be04"}
Sep 08 20:39:46.604906 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.604779    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j" event={"ID":"a86485db-175f-46df-a1ce-e2957d064a3c","Type":"ContainerDied","Data":"35b676cb248e29e5a47692b0c62adfc41e35e0d48c5c46f909a0a619281638e2"}
Sep 08 20:39:46.604906 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.604808    2552 scope.go:117] "RemoveContainer" containerID="4347e8f3e17c01369b6d7114d9e538615aa2f600d21d2f5b06acabe3dd56be04"
Sep 08 20:39:46.607279 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.607120    2552 generic.go:356] "Generic (PLEG): container finished" podID="74bcca5d-b5bf-4bcd-ac0b-8805bcb15906" containerID="450a6b379c7a95f2e29e6612ad660b55601c98c05ba599423fe4586397c05830" exitCode=0
Sep 08 20:39:46.607279 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.607180    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7" event={"ID":"74bcca5d-b5bf-4bcd-ac0b-8805bcb15906","Type":"ContainerDied","Data":"450a6b379c7a95f2e29e6612ad660b55601c98c05ba599423fe4586397c05830"}
Sep 08 20:39:46.607279 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.607186    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7"
Sep 08 20:39:46.607279 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.607220    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7" event={"ID":"74bcca5d-b5bf-4bcd-ac0b-8805bcb15906","Type":"ContainerDied","Data":"09e8dcc2f0a3352b3a3a32526504e4e5ce4c6c639af021cb6b46ba2f1b192a1e"}
Sep 08 20:39:46.616010 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.615989    2552 scope.go:117] "RemoveContainer" containerID="4347e8f3e17c01369b6d7114d9e538615aa2f600d21d2f5b06acabe3dd56be04"
Sep 08 20:39:46.616320 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:39:46.616289    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4347e8f3e17c01369b6d7114d9e538615aa2f600d21d2f5b06acabe3dd56be04\": container with ID starting with 4347e8f3e17c01369b6d7114d9e538615aa2f600d21d2f5b06acabe3dd56be04 not found: ID does not exist" containerID="4347e8f3e17c01369b6d7114d9e538615aa2f600d21d2f5b06acabe3dd56be04"
Sep 08 20:39:46.616410 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.616328    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4347e8f3e17c01369b6d7114d9e538615aa2f600d21d2f5b06acabe3dd56be04"} err="failed to get container status \"4347e8f3e17c01369b6d7114d9e538615aa2f600d21d2f5b06acabe3dd56be04\": rpc error: code = NotFound desc = could not find container \"4347e8f3e17c01369b6d7114d9e538615aa2f600d21d2f5b06acabe3dd56be04\": container with ID starting with 4347e8f3e17c01369b6d7114d9e538615aa2f600d21d2f5b06acabe3dd56be04 not found: ID does not exist"
Sep 08 20:39:46.616410 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.616352    2552 scope.go:117] "RemoveContainer" containerID="450a6b379c7a95f2e29e6612ad660b55601c98c05ba599423fe4586397c05830"
Sep 08 20:39:46.625015 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.624942    2552 scope.go:117] "RemoveContainer" containerID="450a6b379c7a95f2e29e6612ad660b55601c98c05ba599423fe4586397c05830"
Sep 08 20:39:46.625689 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:39:46.625628    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"450a6b379c7a95f2e29e6612ad660b55601c98c05ba599423fe4586397c05830\": container with ID starting with 450a6b379c7a95f2e29e6612ad660b55601c98c05ba599423fe4586397c05830 not found: ID does not exist" containerID="450a6b379c7a95f2e29e6612ad660b55601c98c05ba599423fe4586397c05830"
Sep 08 20:39:46.625689 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.625661    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450a6b379c7a95f2e29e6612ad660b55601c98c05ba599423fe4586397c05830"} err="failed to get container status \"450a6b379c7a95f2e29e6612ad660b55601c98c05ba599423fe4586397c05830\": rpc error: code = NotFound desc = could not find container \"450a6b379c7a95f2e29e6612ad660b55601c98c05ba599423fe4586397c05830\": container with ID starting with 450a6b379c7a95f2e29e6612ad660b55601c98c05ba599423fe4586397c05830 not found: ID does not exist"
Sep 08 20:39:46.660626 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.660596    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a86485db-175f-46df-a1ce-e2957d064a3c-plugins-config-volume\") pod \"a86485db-175f-46df-a1ce-e2957d064a3c\" (UID: \"a86485db-175f-46df-a1ce-e2957d064a3c\") "
Sep 08 20:39:46.660626 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.660633    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dj8bx\" (UniqueName: \"kubernetes.io/projected/a86485db-175f-46df-a1ce-e2957d064a3c-kube-api-access-dj8bx\") pod \"a86485db-175f-46df-a1ce-e2957d064a3c\" (UID: \"a86485db-175f-46df-a1ce-e2957d064a3c\") "
Sep 08 20:39:46.660824 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.660667    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bcca5d-b5bf-4bcd-ac0b-8805bcb15906-plugins-config-volume\") pod \"74bcca5d-b5bf-4bcd-ac0b-8805bcb15906\" (UID: \"74bcca5d-b5bf-4bcd-ac0b-8805bcb15906\") "
Sep 08 20:39:46.660824 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.660722    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mp6z\" (UniqueName: \"kubernetes.io/projected/74bcca5d-b5bf-4bcd-ac0b-8805bcb15906-kube-api-access-9mp6z\") pod \"74bcca5d-b5bf-4bcd-ac0b-8805bcb15906\" (UID: \"74bcca5d-b5bf-4bcd-ac0b-8805bcb15906\") "
Sep 08 20:39:46.661137 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.661110    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a86485db-175f-46df-a1ce-e2957d064a3c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a86485db-175f-46df-a1ce-e2957d064a3c" (UID: "a86485db-175f-46df-a1ce-e2957d064a3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:39:46.661210 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.661110    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74bcca5d-b5bf-4bcd-ac0b-8805bcb15906-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "74bcca5d-b5bf-4bcd-ac0b-8805bcb15906" (UID: "74bcca5d-b5bf-4bcd-ac0b-8805bcb15906"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:39:46.662748 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.662722    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a86485db-175f-46df-a1ce-e2957d064a3c-kube-api-access-dj8bx" (OuterVolumeSpecName: "kube-api-access-dj8bx") pod "a86485db-175f-46df-a1ce-e2957d064a3c" (UID: "a86485db-175f-46df-a1ce-e2957d064a3c"). InnerVolumeSpecName "kube-api-access-dj8bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:39:46.662748 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.662728    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74bcca5d-b5bf-4bcd-ac0b-8805bcb15906-kube-api-access-9mp6z" (OuterVolumeSpecName: "kube-api-access-9mp6z") pod "74bcca5d-b5bf-4bcd-ac0b-8805bcb15906" (UID: "74bcca5d-b5bf-4bcd-ac0b-8805bcb15906"). InnerVolumeSpecName "kube-api-access-9mp6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:39:46.761349 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.761313    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a86485db-175f-46df-a1ce-e2957d064a3c-plugins-config-volume\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:39:46.761349 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.761344    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj8bx\" (UniqueName: \"kubernetes.io/projected/a86485db-175f-46df-a1ce-e2957d064a3c-kube-api-access-dj8bx\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:39:46.761349 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.761357    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bcca5d-b5bf-4bcd-ac0b-8805bcb15906-plugins-config-volume\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:39:46.761573 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.761368    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mp6z\" (UniqueName: \"kubernetes.io/projected/74bcca5d-b5bf-4bcd-ac0b-8805bcb15906-kube-api-access-9mp6z\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:39:46.932285 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.932252    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j"]
Sep 08 20:39:46.936209 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.936184    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4d417ac0-57ff867cf4-8ls7j"]
Sep 08 20:39:46.947008 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.946985    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7"]
Sep 08 20:39:46.952575 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:46.952554    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d417ac0-7995c4c4db-wq7b7"]
Sep 08 20:39:47.225660 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:47.225634    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ee17c3b-ad74-49fa-9a0d-78a69bb7b656" path="/var/lib/kubelet/pods/3ee17c3b-ad74-49fa-9a0d-78a69bb7b656/volumes"
Sep 08 20:39:47.226034 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:47.226020    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74bcca5d-b5bf-4bcd-ac0b-8805bcb15906" path="/var/lib/kubelet/pods/74bcca5d-b5bf-4bcd-ac0b-8805bcb15906/volumes"
Sep 08 20:39:47.226444 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:47.226424    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a86485db-175f-46df-a1ce-e2957d064a3c" path="/var/lib/kubelet/pods/a86485db-175f-46df-a1ce-e2957d064a3c/volumes"
Sep 08 20:39:50.942521 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:50.942490    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"]
Sep 08 20:39:50.942969 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:50.942952    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a86485db-175f-46df-a1ce-e2957d064a3c" containerName="payload-pre-processing"
Sep 08 20:39:50.943038 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:50.942972    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a86485db-175f-46df-a1ce-e2957d064a3c" containerName="payload-pre-processing"
Sep 08 20:39:50.943038 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:50.942992    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ee17c3b-ad74-49fa-9a0d-78a69bb7b656" containerName="revoke-keys"
Sep 08 20:39:50.943038 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:50.943004    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ee17c3b-ad74-49fa-9a0d-78a69bb7b656" containerName="revoke-keys"
Sep 08 20:39:50.943038 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:50.943019    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74bcca5d-b5bf-4bcd-ac0b-8805bcb15906" containerName="payload-processing"
Sep 08 20:39:50.943038 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:50.943028    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="74bcca5d-b5bf-4bcd-ac0b-8805bcb15906" containerName="payload-processing"
Sep 08 20:39:50.943288 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:50.943117    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ee17c3b-ad74-49fa-9a0d-78a69bb7b656" containerName="revoke-keys"
Sep 08 20:39:50.943288 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:50.943132    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="74bcca5d-b5bf-4bcd-ac0b-8805bcb15906" containerName="payload-processing"
Sep 08 20:39:50.943288 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:50.943141    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a86485db-175f-46df-a1ce-e2957d064a3c" containerName="payload-pre-processing"
Sep 08 20:39:50.947896 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:50.947875    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"
Sep 08 20:39:50.950684 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:50.950662    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 20:39:50.951257 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:50.951217    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 20:39:50.951370 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:50.951338    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-z8244\""
Sep 08 20:39:50.951423 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:50.951338    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b2d463b5-serving-cert\""
Sep 08 20:39:50.951492 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:50.951467    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 20:39:50.958683 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:50.958659    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"]
Sep 08 20:39:51.099012 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.098974    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-openshift-service-ca\") pod \"maas-api-e2e-ait-b2d463b5-6685b959f-q47zj\" (UID: \"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"
Sep 08 20:39:51.099180 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.099034    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-maas-api-tls\") pod \"maas-api-e2e-ait-b2d463b5-6685b959f-q47zj\" (UID: \"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"
Sep 08 20:39:51.099180 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.099058    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvvdd\" (UniqueName: \"kubernetes.io/projected/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-kube-api-access-zvvdd\") pod \"maas-api-e2e-ait-b2d463b5-6685b959f-q47zj\" (UID: \"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"
Sep 08 20:39:51.099180 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.099086    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-metrics-certs\") pod \"maas-api-e2e-ait-b2d463b5-6685b959f-q47zj\" (UID: \"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"
Sep 08 20:39:51.200493 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.200391    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-openshift-service-ca\") pod \"maas-api-e2e-ait-b2d463b5-6685b959f-q47zj\" (UID: \"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"
Sep 08 20:39:51.200669 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.200487    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-maas-api-tls\") pod \"maas-api-e2e-ait-b2d463b5-6685b959f-q47zj\" (UID: \"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"
Sep 08 20:39:51.200669 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.200529    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvvdd\" (UniqueName: \"kubernetes.io/projected/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-kube-api-access-zvvdd\") pod \"maas-api-e2e-ait-b2d463b5-6685b959f-q47zj\" (UID: \"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"
Sep 08 20:39:51.200669 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.200603    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-metrics-certs\") pod \"maas-api-e2e-ait-b2d463b5-6685b959f-q47zj\" (UID: \"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"
Sep 08 20:39:51.201130 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.201102    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-openshift-service-ca\") pod \"maas-api-e2e-ait-b2d463b5-6685b959f-q47zj\" (UID: \"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"
Sep 08 20:39:51.203028 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.203005    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-maas-api-tls\") pod \"maas-api-e2e-ait-b2d463b5-6685b959f-q47zj\" (UID: \"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"
Sep 08 20:39:51.203165 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.203144    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-metrics-certs\") pod \"maas-api-e2e-ait-b2d463b5-6685b959f-q47zj\" (UID: \"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"
Sep 08 20:39:51.212994 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.212969    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvvdd\" (UniqueName: \"kubernetes.io/projected/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-kube-api-access-zvvdd\") pod \"maas-api-e2e-ait-b2d463b5-6685b959f-q47zj\" (UID: \"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"
Sep 08 20:39:51.261141 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.261119    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"
Sep 08 20:39:51.545105 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.545076    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"]
Sep 08 20:39:51.546433 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:39:51.546404    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a68a2e3_e3b4_47dd_b6ee_c877695da2c7.slice/crio-70291aafea4cca7af6e6722d32c1bc0e4ca2d9ce32db0d85f67eee621a54dc38 WatchSource:0}: Error finding container 70291aafea4cca7af6e6722d32c1bc0e4ca2d9ce32db0d85f67eee621a54dc38: Status 404 returned error can't find the container with id 70291aafea4cca7af6e6722d32c1bc0e4ca2d9ce32db0d85f67eee621a54dc38
Sep 08 20:39:51.632468 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.632433    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj" event={"ID":"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7","Type":"ContainerStarted","Data":"70291aafea4cca7af6e6722d32c1bc0e4ca2d9ce32db0d85f67eee621a54dc38"}
Sep 08 20:39:51.764679 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.764646    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd"]
Sep 08 20:39:51.769082 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.769065    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd"
Sep 08 20:39:51.771505 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.771482    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b2d463b5\""
Sep 08 20:39:51.780780 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.780759    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd"]
Sep 08 20:39:51.787933 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.787911    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4"]
Sep 08 20:39:51.791156 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.791137    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4"
Sep 08 20:39:51.793883 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.793815    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b2d463b5-dockercfg-gs5f4\""
Sep 08 20:39:51.803282 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.803259    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4"]
Sep 08 20:39:51.905924 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.905891    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd4c983-ce7c-47c6-be00-a321baa70317-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd\" (UID: \"bfd4c983-ce7c-47c6-be00-a321baa70317\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd"
Sep 08 20:39:51.906116 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.905944    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285cf542-e278-4bb9-9d7a-a9ae762e21ab-plugins-config-volume\") pod \"payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4\" (UID: \"285cf542-e278-4bb9-9d7a-a9ae762e21ab\") " pod="openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4"
Sep 08 20:39:51.906116 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.906095    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74q6z\" (UniqueName: \"kubernetes.io/projected/bfd4c983-ce7c-47c6-be00-a321baa70317-kube-api-access-74q6z\") pod \"payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd\" (UID: \"bfd4c983-ce7c-47c6-be00-a321baa70317\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd"
Sep 08 20:39:51.906251 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:51.906156    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qnfp\" (UniqueName: \"kubernetes.io/projected/285cf542-e278-4bb9-9d7a-a9ae762e21ab-kube-api-access-7qnfp\") pod \"payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4\" (UID: \"285cf542-e278-4bb9-9d7a-a9ae762e21ab\") " pod="openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4"
Sep 08 20:39:52.007529 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:52.007493    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd4c983-ce7c-47c6-be00-a321baa70317-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd\" (UID: \"bfd4c983-ce7c-47c6-be00-a321baa70317\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd"
Sep 08 20:39:52.007961 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:52.007547    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285cf542-e278-4bb9-9d7a-a9ae762e21ab-plugins-config-volume\") pod \"payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4\" (UID: \"285cf542-e278-4bb9-9d7a-a9ae762e21ab\") " pod="openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4"
Sep 08 20:39:52.007961 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:52.007642    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74q6z\" (UniqueName: \"kubernetes.io/projected/bfd4c983-ce7c-47c6-be00-a321baa70317-kube-api-access-74q6z\") pod \"payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd\" (UID: \"bfd4c983-ce7c-47c6-be00-a321baa70317\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd"
Sep 08 20:39:52.007961 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:52.007694    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qnfp\" (UniqueName: \"kubernetes.io/projected/285cf542-e278-4bb9-9d7a-a9ae762e21ab-kube-api-access-7qnfp\") pod \"payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4\" (UID: \"285cf542-e278-4bb9-9d7a-a9ae762e21ab\") " pod="openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4"
Sep 08 20:39:52.008344 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:52.008323    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd4c983-ce7c-47c6-be00-a321baa70317-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd\" (UID: \"bfd4c983-ce7c-47c6-be00-a321baa70317\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd"
Sep 08 20:39:52.008514 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:52.008489    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285cf542-e278-4bb9-9d7a-a9ae762e21ab-plugins-config-volume\") pod \"payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4\" (UID: \"285cf542-e278-4bb9-9d7a-a9ae762e21ab\") " pod="openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4"
Sep 08 20:39:52.015975 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:52.015954    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74q6z\" (UniqueName: \"kubernetes.io/projected/bfd4c983-ce7c-47c6-be00-a321baa70317-kube-api-access-74q6z\") pod \"payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd\" (UID: \"bfd4c983-ce7c-47c6-be00-a321baa70317\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd"
Sep 08 20:39:52.016216 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:52.016192    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qnfp\" (UniqueName: \"kubernetes.io/projected/285cf542-e278-4bb9-9d7a-a9ae762e21ab-kube-api-access-7qnfp\") pod \"payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4\" (UID: \"285cf542-e278-4bb9-9d7a-a9ae762e21ab\") " pod="openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4"
Sep 08 20:39:52.080014 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:52.079930    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd"
Sep 08 20:39:52.101972 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:52.101939    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4"
Sep 08 20:39:52.360338 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:52.360304    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd"]
Sep 08 20:39:52.362458 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:39:52.362428    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd4c983_ce7c_47c6_be00_a321baa70317.slice/crio-da8199be655f44d814563e8cc269dbda08e388f02096e3b40961dd41f7303bcf WatchSource:0}: Error finding container da8199be655f44d814563e8cc269dbda08e388f02096e3b40961dd41f7303bcf: Status 404 returned error can't find the container with id da8199be655f44d814563e8cc269dbda08e388f02096e3b40961dd41f7303bcf
Sep 08 20:39:52.381683 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:52.381625    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4"]
Sep 08 20:39:52.383932 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:39:52.383899    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285cf542_e278_4bb9_9d7a_a9ae762e21ab.slice/crio-16074612e10c6a8567b30c5e8d9b0d3740fbc5b793e52013e47e302e7c57467a WatchSource:0}: Error finding container 16074612e10c6a8567b30c5e8d9b0d3740fbc5b793e52013e47e302e7c57467a: Status 404 returned error can't find the container with id 16074612e10c6a8567b30c5e8d9b0d3740fbc5b793e52013e47e302e7c57467a
Sep 08 20:39:52.638957 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:52.638862    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4" event={"ID":"285cf542-e278-4bb9-9d7a-a9ae762e21ab","Type":"ContainerStarted","Data":"be391199fd22985a2486be607bb78cb119f34a8bf4aaee957418fe5c757c71cb"}
Sep 08 20:39:52.638957 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:52.638926    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4" event={"ID":"285cf542-e278-4bb9-9d7a-a9ae762e21ab","Type":"ContainerStarted","Data":"16074612e10c6a8567b30c5e8d9b0d3740fbc5b793e52013e47e302e7c57467a"}
Sep 08 20:39:52.638957 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:52.638955    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4"
Sep 08 20:39:52.640609 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:52.640566    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd" event={"ID":"bfd4c983-ce7c-47c6-be00-a321baa70317","Type":"ContainerStarted","Data":"49164456006adaacdde775f33c14e35373e1458017cedd53c635c490cb00d985"}
Sep 08 20:39:52.640732 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:52.640626    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd" event={"ID":"bfd4c983-ce7c-47c6-be00-a321baa70317","Type":"ContainerStarted","Data":"da8199be655f44d814563e8cc269dbda08e388f02096e3b40961dd41f7303bcf"}
Sep 08 20:39:52.640732 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:52.640656    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd"
Sep 08 20:39:52.657694 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:52.657643    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4" podStartSLOduration=1.657627555 podStartE2EDuration="1.657627555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:39:52.655734335 +0000 UTC m=+845.907215693" watchObservedRunningTime="2026-09-08 20:39:52.657627555 +0000 UTC m=+845.909108912"
Sep 08 20:39:52.673835 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:52.673776    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd" podStartSLOduration=1.6737560280000001 podStartE2EDuration="1.673756028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:39:52.672886891 +0000 UTC m=+845.924368243" watchObservedRunningTime="2026-09-08 20:39:52.673756028 +0000 UTC m=+845.925237388"
Sep 08 20:39:53.656122 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:53.656014    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj" event={"ID":"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7","Type":"ContainerStarted","Data":"e1866619eca6643d0fd21f64671cd5a08bfbaa70177a85cbbe92ebcd8145d409"}
Sep 08 20:39:53.656553 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:53.656341    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"
Sep 08 20:39:53.675869 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:53.675815    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj" podStartSLOduration=1.843415166 podStartE2EDuration="3.675796187s" totalImagesPullingTime="1.832381021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:39:53.673224434 +0000 UTC m=+846.924705793" watchObservedRunningTime="2026-09-08 20:39:53.675796187 +0000 UTC m=+846.927277548"
Sep 08 20:39:59.668051 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:39:59.668021    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"
Sep 08 20:40:00.395362 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:00.395322    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:40:03.658153 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:03.658123    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd"
Sep 08 20:40:03.658585 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:03.658196    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4"
Sep 08 20:40:21.452130 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.452094    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"]
Sep 08 20:40:21.452566 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.452313    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj" podUID="1a68a2e3-e3b4-47dd-b6ee-c877695da2c7" containerName="maas-api" containerID="cri-o://e1866619eca6643d0fd21f64671cd5a08bfbaa70177a85cbbe92ebcd8145d409" gracePeriod=30
Sep 08 20:40:21.498763 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.498733    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4"]
Sep 08 20:40:21.499019 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.498964    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4" podUID="285cf542-e278-4bb9-9d7a-a9ae762e21ab" containerName="payload-processing" containerID="cri-o://be391199fd22985a2486be607bb78cb119f34a8bf4aaee957418fe5c757c71cb" gracePeriod=30
Sep 08 20:40:21.507110 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.507084    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd"]
Sep 08 20:40:21.507349 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.507324    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd" podUID="bfd4c983-ce7c-47c6-be00-a321baa70317" containerName="payload-pre-processing" containerID="cri-o://49164456006adaacdde775f33c14e35373e1458017cedd53c635c490cb00d985" gracePeriod=30
Sep 08 20:40:21.789420 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.789386    2552 generic.go:356] "Generic (PLEG): container finished" podID="1a68a2e3-e3b4-47dd-b6ee-c877695da2c7" containerID="e1866619eca6643d0fd21f64671cd5a08bfbaa70177a85cbbe92ebcd8145d409" exitCode=0
Sep 08 20:40:21.789550 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.789493    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj" event={"ID":"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7","Type":"ContainerDied","Data":"e1866619eca6643d0fd21f64671cd5a08bfbaa70177a85cbbe92ebcd8145d409"}
Sep 08 20:40:21.791121 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.791092    2552 generic.go:356] "Generic (PLEG): container finished" podID="285cf542-e278-4bb9-9d7a-a9ae762e21ab" containerID="be391199fd22985a2486be607bb78cb119f34a8bf4aaee957418fe5c757c71cb" exitCode=0
Sep 08 20:40:21.791272 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.791225    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4" event={"ID":"285cf542-e278-4bb9-9d7a-a9ae762e21ab","Type":"ContainerDied","Data":"be391199fd22985a2486be607bb78cb119f34a8bf4aaee957418fe5c757c71cb"}
Sep 08 20:40:21.793217 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.793183    2552 generic.go:356] "Generic (PLEG): container finished" podID="bfd4c983-ce7c-47c6-be00-a321baa70317" containerID="49164456006adaacdde775f33c14e35373e1458017cedd53c635c490cb00d985" exitCode=0
Sep 08 20:40:21.793339 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.793269    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd" event={"ID":"bfd4c983-ce7c-47c6-be00-a321baa70317","Type":"ContainerDied","Data":"49164456006adaacdde775f33c14e35373e1458017cedd53c635c490cb00d985"}
Sep 08 20:40:21.800853 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.800828    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4"
Sep 08 20:40:21.812923 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.812905    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"
Sep 08 20:40:21.824723 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.824702    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd"
Sep 08 20:40:21.866233 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.866209    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvvdd\" (UniqueName: \"kubernetes.io/projected/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-kube-api-access-zvvdd\") pod \"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7\" (UID: \"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7\") "
Sep 08 20:40:21.866385 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.866292    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd4c983-ce7c-47c6-be00-a321baa70317-plugins-config-volume\") pod \"bfd4c983-ce7c-47c6-be00-a321baa70317\" (UID: \"bfd4c983-ce7c-47c6-be00-a321baa70317\") "
Sep 08 20:40:21.866385 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.866324    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285cf542-e278-4bb9-9d7a-a9ae762e21ab-plugins-config-volume\") pod \"285cf542-e278-4bb9-9d7a-a9ae762e21ab\" (UID: \"285cf542-e278-4bb9-9d7a-a9ae762e21ab\") "
Sep 08 20:40:21.866385 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.866359    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74q6z\" (UniqueName: \"kubernetes.io/projected/bfd4c983-ce7c-47c6-be00-a321baa70317-kube-api-access-74q6z\") pod \"bfd4c983-ce7c-47c6-be00-a321baa70317\" (UID: \"bfd4c983-ce7c-47c6-be00-a321baa70317\") "
Sep 08 20:40:21.866544 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.866418    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-maas-api-tls\") pod \"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7\" (UID: \"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7\") "
Sep 08 20:40:21.866544 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.866449    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-openshift-service-ca\") pod \"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7\" (UID: \"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7\") "
Sep 08 20:40:21.866544 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.866475    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-metrics-certs\") pod \"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7\" (UID: \"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7\") "
Sep 08 20:40:21.866544 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.866518    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qnfp\" (UniqueName: \"kubernetes.io/projected/285cf542-e278-4bb9-9d7a-a9ae762e21ab-kube-api-access-7qnfp\") pod \"285cf542-e278-4bb9-9d7a-a9ae762e21ab\" (UID: \"285cf542-e278-4bb9-9d7a-a9ae762e21ab\") "
Sep 08 20:40:21.866802 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.866778    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285cf542-e278-4bb9-9d7a-a9ae762e21ab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "285cf542-e278-4bb9-9d7a-a9ae762e21ab" (UID: "285cf542-e278-4bb9-9d7a-a9ae762e21ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:40:21.866900 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.866846    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd4c983-ce7c-47c6-be00-a321baa70317-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bfd4c983-ce7c-47c6-be00-a321baa70317" (UID: "bfd4c983-ce7c-47c6-be00-a321baa70317"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:40:21.866967 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.866886    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1a68a2e3-e3b4-47dd-b6ee-c877695da2c7" (UID: "1a68a2e3-e3b4-47dd-b6ee-c877695da2c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:40:21.868718 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.868691    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd4c983-ce7c-47c6-be00-a321baa70317-kube-api-access-74q6z" (OuterVolumeSpecName: "kube-api-access-74q6z") pod "bfd4c983-ce7c-47c6-be00-a321baa70317" (UID: "bfd4c983-ce7c-47c6-be00-a321baa70317"). InnerVolumeSpecName "kube-api-access-74q6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:40:21.868847 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.868823    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1a68a2e3-e3b4-47dd-b6ee-c877695da2c7" (UID: "1a68a2e3-e3b4-47dd-b6ee-c877695da2c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 20:40:21.868952 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.868933    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-kube-api-access-zvvdd" (OuterVolumeSpecName: "kube-api-access-zvvdd") pod "1a68a2e3-e3b4-47dd-b6ee-c877695da2c7" (UID: "1a68a2e3-e3b4-47dd-b6ee-c877695da2c7"). InnerVolumeSpecName "kube-api-access-zvvdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:40:21.869031 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.869013    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285cf542-e278-4bb9-9d7a-a9ae762e21ab-kube-api-access-7qnfp" (OuterVolumeSpecName: "kube-api-access-7qnfp") pod "285cf542-e278-4bb9-9d7a-a9ae762e21ab" (UID: "285cf542-e278-4bb9-9d7a-a9ae762e21ab"). InnerVolumeSpecName "kube-api-access-7qnfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:40:21.869156 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.869137    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1a68a2e3-e3b4-47dd-b6ee-c877695da2c7" (UID: "1a68a2e3-e3b4-47dd-b6ee-c877695da2c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 20:40:21.967585 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.967497    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-maas-api-tls\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:40:21.967585 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.967529    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-openshift-service-ca\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:40:21.967585 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.967539    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-metrics-certs\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:40:21.967585 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.967547    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qnfp\" (UniqueName: \"kubernetes.io/projected/285cf542-e278-4bb9-9d7a-a9ae762e21ab-kube-api-access-7qnfp\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:40:21.967585 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.967556    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvvdd\" (UniqueName: \"kubernetes.io/projected/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7-kube-api-access-zvvdd\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:40:21.967585 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.967565    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd4c983-ce7c-47c6-be00-a321baa70317-plugins-config-volume\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:40:21.967585 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.967575    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285cf542-e278-4bb9-9d7a-a9ae762e21ab-plugins-config-volume\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:40:21.967585 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:21.967583    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74q6z\" (UniqueName: \"kubernetes.io/projected/bfd4c983-ce7c-47c6-be00-a321baa70317-kube-api-access-74q6z\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:40:22.395106 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:22.395070    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:40:22.798724 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:22.798632    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj" event={"ID":"1a68a2e3-e3b4-47dd-b6ee-c877695da2c7","Type":"ContainerDied","Data":"70291aafea4cca7af6e6722d32c1bc0e4ca2d9ce32db0d85f67eee621a54dc38"}
Sep 08 20:40:22.798724 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:22.798649    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"
Sep 08 20:40:22.798724 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:22.798722    2552 scope.go:117] "RemoveContainer" containerID="e1866619eca6643d0fd21f64671cd5a08bfbaa70177a85cbbe92ebcd8145d409"
Sep 08 20:40:22.800185 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:22.800169    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4"
Sep 08 20:40:22.800481 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:22.800196    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4" event={"ID":"285cf542-e278-4bb9-9d7a-a9ae762e21ab","Type":"ContainerDied","Data":"16074612e10c6a8567b30c5e8d9b0d3740fbc5b793e52013e47e302e7c57467a"}
Sep 08 20:40:22.801742 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:22.801718    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd"
Sep 08 20:40:22.801742 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:22.801729    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd" event={"ID":"bfd4c983-ce7c-47c6-be00-a321baa70317","Type":"ContainerDied","Data":"da8199be655f44d814563e8cc269dbda08e388f02096e3b40961dd41f7303bcf"}
Sep 08 20:40:22.809415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:22.809349    2552 scope.go:117] "RemoveContainer" containerID="be391199fd22985a2486be607bb78cb119f34a8bf4aaee957418fe5c757c71cb"
Sep 08 20:40:22.817760 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:22.817743    2552 scope.go:117] "RemoveContainer" containerID="49164456006adaacdde775f33c14e35373e1458017cedd53c635c490cb00d985"
Sep 08 20:40:22.825526 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:22.825504    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"]
Sep 08 20:40:22.830057 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:22.830036    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b2d463b5-6685b959f-q47zj"]
Sep 08 20:40:22.840203 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:22.840180    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd"]
Sep 08 20:40:22.843808 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:22.843788    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2d463b5-5ddbbf55f4-6bbcd"]
Sep 08 20:40:22.853855 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:22.853835    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4"]
Sep 08 20:40:22.858708 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:22.858689    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b2d463b5-7d9bc9d7b7-lpwd4"]
Sep 08 20:40:23.225192 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:23.225155    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a68a2e3-e3b4-47dd-b6ee-c877695da2c7" path="/var/lib/kubelet/pods/1a68a2e3-e3b4-47dd-b6ee-c877695da2c7/volumes"
Sep 08 20:40:23.225640 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:23.225620    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="285cf542-e278-4bb9-9d7a-a9ae762e21ab" path="/var/lib/kubelet/pods/285cf542-e278-4bb9-9d7a-a9ae762e21ab/volumes"
Sep 08 20:40:23.226005 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:23.225976    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfd4c983-ce7c-47c6-be00-a321baa70317" path="/var/lib/kubelet/pods/bfd4c983-ce7c-47c6-be00-a321baa70317/volumes"
Sep 08 20:40:27.385156 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.385127    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"]
Sep 08 20:40:27.385540 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.385476    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfd4c983-ce7c-47c6-be00-a321baa70317" containerName="payload-pre-processing"
Sep 08 20:40:27.385540 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.385492    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfd4c983-ce7c-47c6-be00-a321baa70317" containerName="payload-pre-processing"
Sep 08 20:40:27.385540 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.385501    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a68a2e3-e3b4-47dd-b6ee-c877695da2c7" containerName="maas-api"
Sep 08 20:40:27.385540 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.385506    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a68a2e3-e3b4-47dd-b6ee-c877695da2c7" containerName="maas-api"
Sep 08 20:40:27.385540 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.385517    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="285cf542-e278-4bb9-9d7a-a9ae762e21ab" containerName="payload-processing"
Sep 08 20:40:27.385540 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.385523    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="285cf542-e278-4bb9-9d7a-a9ae762e21ab" containerName="payload-processing"
Sep 08 20:40:27.385718 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.385580    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfd4c983-ce7c-47c6-be00-a321baa70317" containerName="payload-pre-processing"
Sep 08 20:40:27.385718 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.385588    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a68a2e3-e3b4-47dd-b6ee-c877695da2c7" containerName="maas-api"
Sep 08 20:40:27.385718 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.385594    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="285cf542-e278-4bb9-9d7a-a9ae762e21ab" containerName="payload-processing"
Sep 08 20:40:27.391623 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.391603    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"
Sep 08 20:40:27.393900 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.393874    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-ad269717-serving-cert\""
Sep 08 20:40:27.394011 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.393918    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 20:40:27.394011 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.393918    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 20:40:27.394669 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.394646    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-z8244\""
Sep 08 20:40:27.394790 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.394680    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 20:40:27.398342 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.398323    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"]
Sep 08 20:40:27.525391 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.525363    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/209ab54d-4c4d-486b-b684-2c2efe416097-metrics-certs\") pod \"maas-api-e2e-derive-ad269717-6b97984987-km9rq\" (UID: \"209ab54d-4c4d-486b-b684-2c2efe416097\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"
Sep 08 20:40:27.525540 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.525400    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg2xh\" (UniqueName: \"kubernetes.io/projected/209ab54d-4c4d-486b-b684-2c2efe416097-kube-api-access-zg2xh\") pod \"maas-api-e2e-derive-ad269717-6b97984987-km9rq\" (UID: \"209ab54d-4c4d-486b-b684-2c2efe416097\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"
Sep 08 20:40:27.525540 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.525426    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/209ab54d-4c4d-486b-b684-2c2efe416097-openshift-service-ca\") pod \"maas-api-e2e-derive-ad269717-6b97984987-km9rq\" (UID: \"209ab54d-4c4d-486b-b684-2c2efe416097\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"
Sep 08 20:40:27.525636 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.525533    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/209ab54d-4c4d-486b-b684-2c2efe416097-maas-api-tls\") pod \"maas-api-e2e-derive-ad269717-6b97984987-km9rq\" (UID: \"209ab54d-4c4d-486b-b684-2c2efe416097\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"
Sep 08 20:40:27.626806 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.626776    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/209ab54d-4c4d-486b-b684-2c2efe416097-maas-api-tls\") pod \"maas-api-e2e-derive-ad269717-6b97984987-km9rq\" (UID: \"209ab54d-4c4d-486b-b684-2c2efe416097\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"
Sep 08 20:40:27.627006 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.626839    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/209ab54d-4c4d-486b-b684-2c2efe416097-metrics-certs\") pod \"maas-api-e2e-derive-ad269717-6b97984987-km9rq\" (UID: \"209ab54d-4c4d-486b-b684-2c2efe416097\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"
Sep 08 20:40:27.627006 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.626866    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg2xh\" (UniqueName: \"kubernetes.io/projected/209ab54d-4c4d-486b-b684-2c2efe416097-kube-api-access-zg2xh\") pod \"maas-api-e2e-derive-ad269717-6b97984987-km9rq\" (UID: \"209ab54d-4c4d-486b-b684-2c2efe416097\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"
Sep 08 20:40:27.627006 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.626892    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/209ab54d-4c4d-486b-b684-2c2efe416097-openshift-service-ca\") pod \"maas-api-e2e-derive-ad269717-6b97984987-km9rq\" (UID: \"209ab54d-4c4d-486b-b684-2c2efe416097\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"
Sep 08 20:40:27.627550 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.627525    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/209ab54d-4c4d-486b-b684-2c2efe416097-openshift-service-ca\") pod \"maas-api-e2e-derive-ad269717-6b97984987-km9rq\" (UID: \"209ab54d-4c4d-486b-b684-2c2efe416097\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"
Sep 08 20:40:27.629379 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.629353    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/209ab54d-4c4d-486b-b684-2c2efe416097-metrics-certs\") pod \"maas-api-e2e-derive-ad269717-6b97984987-km9rq\" (UID: \"209ab54d-4c4d-486b-b684-2c2efe416097\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"
Sep 08 20:40:27.629478 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.629364    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/209ab54d-4c4d-486b-b684-2c2efe416097-maas-api-tls\") pod \"maas-api-e2e-derive-ad269717-6b97984987-km9rq\" (UID: \"209ab54d-4c4d-486b-b684-2c2efe416097\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"
Sep 08 20:40:27.636275 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.636197    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg2xh\" (UniqueName: \"kubernetes.io/projected/209ab54d-4c4d-486b-b684-2c2efe416097-kube-api-access-zg2xh\") pod \"maas-api-e2e-derive-ad269717-6b97984987-km9rq\" (UID: \"209ab54d-4c4d-486b-b684-2c2efe416097\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"
Sep 08 20:40:27.704995 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.704963    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"
Sep 08 20:40:27.974957 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.974866    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6"]
Sep 08 20:40:27.979048 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.979026    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6"
Sep 08 20:40:27.981213 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.981184    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"]
Sep 08 20:40:27.981513 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.981488    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ad269717\""
Sep 08 20:40:27.981643 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.981492    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ad269717-dockercfg-tqpf9\""
Sep 08 20:40:27.984666 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:40:27.984645    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209ab54d_4c4d_486b_b684_2c2efe416097.slice/crio-a6ff9f2b83dee3aa1d9330faa9ce7f359c29df9e59c4b70723ee2862e7af2355 WatchSource:0}: Error finding container a6ff9f2b83dee3aa1d9330faa9ce7f359c29df9e59c4b70723ee2862e7af2355: Status 404 returned error can't find the container with id a6ff9f2b83dee3aa1d9330faa9ce7f359c29df9e59c4b70723ee2862e7af2355
Sep 08 20:40:27.988687 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:27.988666    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6"]
Sep 08 20:40:28.005566 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.005538    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7"]
Sep 08 20:40:28.009220 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.009204    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7"
Sep 08 20:40:28.023827 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.023804    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7"]
Sep 08 20:40:28.131644 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.131614    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdvdb\" (UniqueName: \"kubernetes.io/projected/42de4f00-0da1-4bf0-affd-8b3e872df25e-kube-api-access-mdvdb\") pod \"payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6\" (UID: \"42de4f00-0da1-4bf0-affd-8b3e872df25e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6"
Sep 08 20:40:28.131644 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.131650    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42de4f00-0da1-4bf0-affd-8b3e872df25e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6\" (UID: \"42de4f00-0da1-4bf0-affd-8b3e872df25e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6"
Sep 08 20:40:28.131846 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.131736    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc7bp\" (UniqueName: \"kubernetes.io/projected/b12827ac-6f2d-4040-b9c7-7a1b941eb6bb-kube-api-access-vc7bp\") pod \"payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7\" (UID: \"b12827ac-6f2d-4040-b9c7-7a1b941eb6bb\") " pod="openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7"
Sep 08 20:40:28.131846 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.131815    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12827ac-6f2d-4040-b9c7-7a1b941eb6bb-plugins-config-volume\") pod \"payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7\" (UID: \"b12827ac-6f2d-4040-b9c7-7a1b941eb6bb\") " pod="openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7"
Sep 08 20:40:28.232565 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.232483    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc7bp\" (UniqueName: \"kubernetes.io/projected/b12827ac-6f2d-4040-b9c7-7a1b941eb6bb-kube-api-access-vc7bp\") pod \"payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7\" (UID: \"b12827ac-6f2d-4040-b9c7-7a1b941eb6bb\") " pod="openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7"
Sep 08 20:40:28.232565 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.232545    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12827ac-6f2d-4040-b9c7-7a1b941eb6bb-plugins-config-volume\") pod \"payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7\" (UID: \"b12827ac-6f2d-4040-b9c7-7a1b941eb6bb\") " pod="openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7"
Sep 08 20:40:28.232742 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.232580    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdvdb\" (UniqueName: \"kubernetes.io/projected/42de4f00-0da1-4bf0-affd-8b3e872df25e-kube-api-access-mdvdb\") pod \"payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6\" (UID: \"42de4f00-0da1-4bf0-affd-8b3e872df25e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6"
Sep 08 20:40:28.232742 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.232598    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42de4f00-0da1-4bf0-affd-8b3e872df25e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6\" (UID: \"42de4f00-0da1-4bf0-affd-8b3e872df25e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6"
Sep 08 20:40:28.233357 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.233336    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42de4f00-0da1-4bf0-affd-8b3e872df25e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6\" (UID: \"42de4f00-0da1-4bf0-affd-8b3e872df25e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6"
Sep 08 20:40:28.233441 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.233336    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12827ac-6f2d-4040-b9c7-7a1b941eb6bb-plugins-config-volume\") pod \"payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7\" (UID: \"b12827ac-6f2d-4040-b9c7-7a1b941eb6bb\") " pod="openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7"
Sep 08 20:40:28.241125 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.241097    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdvdb\" (UniqueName: \"kubernetes.io/projected/42de4f00-0da1-4bf0-affd-8b3e872df25e-kube-api-access-mdvdb\") pod \"payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6\" (UID: \"42de4f00-0da1-4bf0-affd-8b3e872df25e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6"
Sep 08 20:40:28.241233 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.241150    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc7bp\" (UniqueName: \"kubernetes.io/projected/b12827ac-6f2d-4040-b9c7-7a1b941eb6bb-kube-api-access-vc7bp\") pod \"payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7\" (UID: \"b12827ac-6f2d-4040-b9c7-7a1b941eb6bb\") " pod="openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7"
Sep 08 20:40:28.299784 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.299756    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6"
Sep 08 20:40:28.319288 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.319262    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7"
Sep 08 20:40:28.573645 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.573618    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6"]
Sep 08 20:40:28.575709 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:40:28.575684    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42de4f00_0da1_4bf0_affd_8b3e872df25e.slice/crio-5db0f36e8801af2aeba07b873722b0770595eaa5234c413d3227eb7705b6fb93 WatchSource:0}: Error finding container 5db0f36e8801af2aeba07b873722b0770595eaa5234c413d3227eb7705b6fb93: Status 404 returned error can't find the container with id 5db0f36e8801af2aeba07b873722b0770595eaa5234c413d3227eb7705b6fb93
Sep 08 20:40:28.599097 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.599072    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7"]
Sep 08 20:40:28.609312 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:40:28.609289    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12827ac_6f2d_4040_b9c7_7a1b941eb6bb.slice/crio-0540f58dcd755814078ddcb6c2737b913026391361d576332d8d16a164975dbe WatchSource:0}: Error finding container 0540f58dcd755814078ddcb6c2737b913026391361d576332d8d16a164975dbe: Status 404 returned error can't find the container with id 0540f58dcd755814078ddcb6c2737b913026391361d576332d8d16a164975dbe
Sep 08 20:40:28.835078 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.835045    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7" event={"ID":"b12827ac-6f2d-4040-b9c7-7a1b941eb6bb","Type":"ContainerStarted","Data":"93a3482c2df8ee5b71fde4b8e8ab253e75f28c915db06b4b538775105d1defc6"}
Sep 08 20:40:28.835257 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.835106    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7" event={"ID":"b12827ac-6f2d-4040-b9c7-7a1b941eb6bb","Type":"ContainerStarted","Data":"0540f58dcd755814078ddcb6c2737b913026391361d576332d8d16a164975dbe"}
Sep 08 20:40:28.835257 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.835217    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7"
Sep 08 20:40:28.836793 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.836771    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6" event={"ID":"42de4f00-0da1-4bf0-affd-8b3e872df25e","Type":"ContainerStarted","Data":"31e5999b616999135151fb874599edf456005575c1772d0507e972928ae844ef"}
Sep 08 20:40:28.836967 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.836946    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6"
Sep 08 20:40:28.837076 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.836979    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6" event={"ID":"42de4f00-0da1-4bf0-affd-8b3e872df25e","Type":"ContainerStarted","Data":"5db0f36e8801af2aeba07b873722b0770595eaa5234c413d3227eb7705b6fb93"}
Sep 08 20:40:28.838627 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.838601    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq" event={"ID":"209ab54d-4c4d-486b-b684-2c2efe416097","Type":"ContainerStarted","Data":"6baa19528bfe39c4db3e42cb69cac87b6817a0ef7508fe854b2017d277569e9b"}
Sep 08 20:40:28.838717 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.838644    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq" event={"ID":"209ab54d-4c4d-486b-b684-2c2efe416097","Type":"ContainerStarted","Data":"a6ff9f2b83dee3aa1d9330faa9ce7f359c29df9e59c4b70723ee2862e7af2355"}
Sep 08 20:40:28.838717 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.838670    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"
Sep 08 20:40:28.852840 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.852795    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7" podStartSLOduration=1.8527770829999999 podStartE2EDuration="1.852777083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:40:28.851728684 +0000 UTC m=+882.103210045" watchObservedRunningTime="2026-09-08 20:40:28.852777083 +0000 UTC m=+882.104258442"
Sep 08 20:40:28.872153 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.872098    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq" podStartSLOduration=1.107940666 podStartE2EDuration="1.872080486s" totalImagesPullingTime="764.13982ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:40:28.871509618 +0000 UTC m=+882.122991031" watchObservedRunningTime="2026-09-08 20:40:28.872080486 +0000 UTC m=+882.123561849"
Sep 08 20:40:28.893456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:28.893399    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6" podStartSLOduration=1.8933816509999999 podStartE2EDuration="1.893381651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:40:28.891737611 +0000 UTC m=+882.143218970" watchObservedRunningTime="2026-09-08 20:40:28.893381651 +0000 UTC m=+882.144863011"
Sep 08 20:40:34.849679 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:34.849651    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"
Sep 08 20:40:37.991359 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:37.991325    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:40:39.846025 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:39.845937    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7"
Sep 08 20:40:39.846474 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:39.846067    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6"
Sep 08 20:40:55.495704 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:55.495670    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:40:57.798026 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:57.797994    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"]
Sep 08 20:40:57.798483 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:57.798262    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq" podUID="209ab54d-4c4d-486b-b684-2c2efe416097" containerName="maas-api" containerID="cri-o://6baa19528bfe39c4db3e42cb69cac87b6817a0ef7508fe854b2017d277569e9b" gracePeriod=30
Sep 08 20:40:57.848787 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:57.848749    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7"]
Sep 08 20:40:57.849046 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:57.849022    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7" podUID="b12827ac-6f2d-4040-b9c7-7a1b941eb6bb" containerName="payload-processing" containerID="cri-o://93a3482c2df8ee5b71fde4b8e8ab253e75f28c915db06b4b538775105d1defc6" gracePeriod=30
Sep 08 20:40:57.857950 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:57.857915    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6"]
Sep 08 20:40:57.858176 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:57.858148    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6" podUID="42de4f00-0da1-4bf0-affd-8b3e872df25e" containerName="payload-pre-processing" containerID="cri-o://31e5999b616999135151fb874599edf456005575c1772d0507e972928ae844ef" gracePeriod=30
Sep 08 20:40:57.974483 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:57.974456    2552 generic.go:356] "Generic (PLEG): container finished" podID="b12827ac-6f2d-4040-b9c7-7a1b941eb6bb" containerID="93a3482c2df8ee5b71fde4b8e8ab253e75f28c915db06b4b538775105d1defc6" exitCode=0
Sep 08 20:40:57.974602 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:57.974530    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7" event={"ID":"b12827ac-6f2d-4040-b9c7-7a1b941eb6bb","Type":"ContainerDied","Data":"93a3482c2df8ee5b71fde4b8e8ab253e75f28c915db06b4b538775105d1defc6"}
Sep 08 20:40:57.976438 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:57.976408    2552 generic.go:356] "Generic (PLEG): container finished" podID="42de4f00-0da1-4bf0-affd-8b3e872df25e" containerID="31e5999b616999135151fb874599edf456005575c1772d0507e972928ae844ef" exitCode=0
Sep 08 20:40:57.976541 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:57.976523    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6" event={"ID":"42de4f00-0da1-4bf0-affd-8b3e872df25e","Type":"ContainerDied","Data":"31e5999b616999135151fb874599edf456005575c1772d0507e972928ae844ef"}
Sep 08 20:40:57.978443 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:57.978412    2552 generic.go:356] "Generic (PLEG): container finished" podID="209ab54d-4c4d-486b-b684-2c2efe416097" containerID="6baa19528bfe39c4db3e42cb69cac87b6817a0ef7508fe854b2017d277569e9b" exitCode=0
Sep 08 20:40:57.978556 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:57.978538    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq" event={"ID":"209ab54d-4c4d-486b-b684-2c2efe416097","Type":"ContainerDied","Data":"6baa19528bfe39c4db3e42cb69cac87b6817a0ef7508fe854b2017d277569e9b"}
Sep 08 20:40:58.109630 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.109603    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"
Sep 08 20:40:58.164760 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.164734    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6"
Sep 08 20:40:58.167488 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.167470    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7"
Sep 08 20:40:58.187814 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.187788    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/209ab54d-4c4d-486b-b684-2c2efe416097-metrics-certs\") pod \"209ab54d-4c4d-486b-b684-2c2efe416097\" (UID: \"209ab54d-4c4d-486b-b684-2c2efe416097\") "
Sep 08 20:40:58.187918 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.187829    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12827ac-6f2d-4040-b9c7-7a1b941eb6bb-plugins-config-volume\") pod \"b12827ac-6f2d-4040-b9c7-7a1b941eb6bb\" (UID: \"b12827ac-6f2d-4040-b9c7-7a1b941eb6bb\") "
Sep 08 20:40:58.187974 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.187952    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/209ab54d-4c4d-486b-b684-2c2efe416097-openshift-service-ca\") pod \"209ab54d-4c4d-486b-b684-2c2efe416097\" (UID: \"209ab54d-4c4d-486b-b684-2c2efe416097\") "
Sep 08 20:40:58.188029 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.187988    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vc7bp\" (UniqueName: \"kubernetes.io/projected/b12827ac-6f2d-4040-b9c7-7a1b941eb6bb-kube-api-access-vc7bp\") pod \"b12827ac-6f2d-4040-b9c7-7a1b941eb6bb\" (UID: \"b12827ac-6f2d-4040-b9c7-7a1b941eb6bb\") "
Sep 08 20:40:58.188087 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.188072    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/209ab54d-4c4d-486b-b684-2c2efe416097-maas-api-tls\") pod \"209ab54d-4c4d-486b-b684-2c2efe416097\" (UID: \"209ab54d-4c4d-486b-b684-2c2efe416097\") "
Sep 08 20:40:58.188144 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.188131    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zg2xh\" (UniqueName: \"kubernetes.io/projected/209ab54d-4c4d-486b-b684-2c2efe416097-kube-api-access-zg2xh\") pod \"209ab54d-4c4d-486b-b684-2c2efe416097\" (UID: \"209ab54d-4c4d-486b-b684-2c2efe416097\") "
Sep 08 20:40:58.188197 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.188162    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42de4f00-0da1-4bf0-affd-8b3e872df25e-plugins-config-volume\") pod \"42de4f00-0da1-4bf0-affd-8b3e872df25e\" (UID: \"42de4f00-0da1-4bf0-affd-8b3e872df25e\") "
Sep 08 20:40:58.188282 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.188203    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdvdb\" (UniqueName: \"kubernetes.io/projected/42de4f00-0da1-4bf0-affd-8b3e872df25e-kube-api-access-mdvdb\") pod \"42de4f00-0da1-4bf0-affd-8b3e872df25e\" (UID: \"42de4f00-0da1-4bf0-affd-8b3e872df25e\") "
Sep 08 20:40:58.188282 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.188254    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b12827ac-6f2d-4040-b9c7-7a1b941eb6bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b12827ac-6f2d-4040-b9c7-7a1b941eb6bb" (UID: "b12827ac-6f2d-4040-b9c7-7a1b941eb6bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:40:58.188382 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.188319    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/209ab54d-4c4d-486b-b684-2c2efe416097-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "209ab54d-4c4d-486b-b684-2c2efe416097" (UID: "209ab54d-4c4d-486b-b684-2c2efe416097"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:40:58.188557 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.188531    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12827ac-6f2d-4040-b9c7-7a1b941eb6bb-plugins-config-volume\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:40:58.188626 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.188561    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/209ab54d-4c4d-486b-b684-2c2efe416097-openshift-service-ca\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:40:58.188893 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.188859    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42de4f00-0da1-4bf0-affd-8b3e872df25e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42de4f00-0da1-4bf0-affd-8b3e872df25e" (UID: "42de4f00-0da1-4bf0-affd-8b3e872df25e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:40:58.190385 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.190359    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12827ac-6f2d-4040-b9c7-7a1b941eb6bb-kube-api-access-vc7bp" (OuterVolumeSpecName: "kube-api-access-vc7bp") pod "b12827ac-6f2d-4040-b9c7-7a1b941eb6bb" (UID: "b12827ac-6f2d-4040-b9c7-7a1b941eb6bb"). InnerVolumeSpecName "kube-api-access-vc7bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:40:58.190734 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.190708    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/209ab54d-4c4d-486b-b684-2c2efe416097-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "209ab54d-4c4d-486b-b684-2c2efe416097" (UID: "209ab54d-4c4d-486b-b684-2c2efe416097"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 20:40:58.190821 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.190799    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42de4f00-0da1-4bf0-affd-8b3e872df25e-kube-api-access-mdvdb" (OuterVolumeSpecName: "kube-api-access-mdvdb") pod "42de4f00-0da1-4bf0-affd-8b3e872df25e" (UID: "42de4f00-0da1-4bf0-affd-8b3e872df25e"). InnerVolumeSpecName "kube-api-access-mdvdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:40:58.190884 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.190809    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209ab54d-4c4d-486b-b684-2c2efe416097-kube-api-access-zg2xh" (OuterVolumeSpecName: "kube-api-access-zg2xh") pod "209ab54d-4c4d-486b-b684-2c2efe416097" (UID: "209ab54d-4c4d-486b-b684-2c2efe416097"). InnerVolumeSpecName "kube-api-access-zg2xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:40:58.190956 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.190931    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/209ab54d-4c4d-486b-b684-2c2efe416097-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "209ab54d-4c4d-486b-b684-2c2efe416097" (UID: "209ab54d-4c4d-486b-b684-2c2efe416097"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 20:40:58.289147 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.289116    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc7bp\" (UniqueName: \"kubernetes.io/projected/b12827ac-6f2d-4040-b9c7-7a1b941eb6bb-kube-api-access-vc7bp\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:40:58.289147 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.289142    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/209ab54d-4c4d-486b-b684-2c2efe416097-maas-api-tls\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:40:58.289147 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.289154    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg2xh\" (UniqueName: \"kubernetes.io/projected/209ab54d-4c4d-486b-b684-2c2efe416097-kube-api-access-zg2xh\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:40:58.289449 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.289169    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42de4f00-0da1-4bf0-affd-8b3e872df25e-plugins-config-volume\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:40:58.289449 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.289182    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdvdb\" (UniqueName: \"kubernetes.io/projected/42de4f00-0da1-4bf0-affd-8b3e872df25e-kube-api-access-mdvdb\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:40:58.289449 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.289195    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/209ab54d-4c4d-486b-b684-2c2efe416097-metrics-certs\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:40:58.984220 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.984180    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"
Sep 08 20:40:58.984220 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.984212    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq" event={"ID":"209ab54d-4c4d-486b-b684-2c2efe416097","Type":"ContainerDied","Data":"a6ff9f2b83dee3aa1d9330faa9ce7f359c29df9e59c4b70723ee2862e7af2355"}
Sep 08 20:40:58.984731 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.984298    2552 scope.go:117] "RemoveContainer" containerID="6baa19528bfe39c4db3e42cb69cac87b6817a0ef7508fe854b2017d277569e9b"
Sep 08 20:40:58.985943 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.985921    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7"
Sep 08 20:40:58.986059 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.985924    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7" event={"ID":"b12827ac-6f2d-4040-b9c7-7a1b941eb6bb","Type":"ContainerDied","Data":"0540f58dcd755814078ddcb6c2737b913026391361d576332d8d16a164975dbe"}
Sep 08 20:40:58.991726 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.991699    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6" event={"ID":"42de4f00-0da1-4bf0-affd-8b3e872df25e","Type":"ContainerDied","Data":"5db0f36e8801af2aeba07b873722b0770595eaa5234c413d3227eb7705b6fb93"}
Sep 08 20:40:58.991859 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.991730    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6"
Sep 08 20:40:58.995963 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:58.995943    2552 scope.go:117] "RemoveContainer" containerID="93a3482c2df8ee5b71fde4b8e8ab253e75f28c915db06b4b538775105d1defc6"
Sep 08 20:40:59.009891 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:59.009874    2552 scope.go:117] "RemoveContainer" containerID="31e5999b616999135151fb874599edf456005575c1772d0507e972928ae844ef"
Sep 08 20:40:59.020518 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:59.020495    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7"]
Sep 08 20:40:59.024444 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:59.024419    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ad269717-ccf79b85-tpqg7"]
Sep 08 20:40:59.038473 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:59.038452    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"]
Sep 08 20:40:59.048201 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:59.048178    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ad269717-6b97984987-km9rq"]
Sep 08 20:40:59.058672 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:59.058653    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6"]
Sep 08 20:40:59.061516 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:59.061497    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ad269717-7ddcd4fd54-lrfq6"]
Sep 08 20:40:59.224966 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:59.224938    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="209ab54d-4c4d-486b-b684-2c2efe416097" path="/var/lib/kubelet/pods/209ab54d-4c4d-486b-b684-2c2efe416097/volumes"
Sep 08 20:40:59.225308 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:59.225295    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42de4f00-0da1-4bf0-affd-8b3e872df25e" path="/var/lib/kubelet/pods/42de4f00-0da1-4bf0-affd-8b3e872df25e/volumes"
Sep 08 20:40:59.225611 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:40:59.225599    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b12827ac-6f2d-4040-b9c7-7a1b941eb6bb" path="/var/lib/kubelet/pods/b12827ac-6f2d-4040-b9c7-7a1b941eb6bb/volumes"
Sep 08 20:41:12.096059 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:12.096010    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:41:19.195268 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.195216    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"]
Sep 08 20:41:19.195629 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.195592    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42de4f00-0da1-4bf0-affd-8b3e872df25e" containerName="payload-pre-processing"
Sep 08 20:41:19.195629 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.195607    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="42de4f00-0da1-4bf0-affd-8b3e872df25e" containerName="payload-pre-processing"
Sep 08 20:41:19.195629 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.195624    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b12827ac-6f2d-4040-b9c7-7a1b941eb6bb" containerName="payload-processing"
Sep 08 20:41:19.195629 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.195630    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b12827ac-6f2d-4040-b9c7-7a1b941eb6bb" containerName="payload-processing"
Sep 08 20:41:19.195769 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.195636    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="209ab54d-4c4d-486b-b684-2c2efe416097" containerName="maas-api"
Sep 08 20:41:19.195769 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.195642    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="209ab54d-4c4d-486b-b684-2c2efe416097" containerName="maas-api"
Sep 08 20:41:19.195769 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.195709    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="42de4f00-0da1-4bf0-affd-8b3e872df25e" containerName="payload-pre-processing"
Sep 08 20:41:19.195769 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.195720    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b12827ac-6f2d-4040-b9c7-7a1b941eb6bb" containerName="payload-processing"
Sep 08 20:41:19.195769 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.195727    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="209ab54d-4c4d-486b-b684-2c2efe416097" containerName="maas-api"
Sep 08 20:41:19.198735 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.198712    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"
Sep 08 20:41:19.201560 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.201533    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-b433c5-serving-cert\""
Sep 08 20:41:19.202398 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.202377    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 20:41:19.202577 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.202551    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 20:41:19.202737 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.202716    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 20:41:19.202879 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.202853    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-z8244\""
Sep 08 20:41:19.208870 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.208844    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"]
Sep 08 20:41:19.269021 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.268988    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-metrics-certs\") pod \"maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl\" (UID: \"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"
Sep 08 20:41:19.269197 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.269029    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhtnm\" (UniqueName: \"kubernetes.io/projected/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-kube-api-access-xhtnm\") pod \"maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl\" (UID: \"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"
Sep 08 20:41:19.269197 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.269110    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-maas-api-tls\") pod \"maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl\" (UID: \"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"
Sep 08 20:41:19.269197 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.269149    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-openshift-service-ca\") pod \"maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl\" (UID: \"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"
Sep 08 20:41:19.370679 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.370639    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-metrics-certs\") pod \"maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl\" (UID: \"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"
Sep 08 20:41:19.370858 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.370695    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhtnm\" (UniqueName: \"kubernetes.io/projected/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-kube-api-access-xhtnm\") pod \"maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl\" (UID: \"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"
Sep 08 20:41:19.370858 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.370742    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-maas-api-tls\") pod \"maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl\" (UID: \"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"
Sep 08 20:41:19.370858 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.370769    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-openshift-service-ca\") pod \"maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl\" (UID: \"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"
Sep 08 20:41:19.371421 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.371401    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-openshift-service-ca\") pod \"maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl\" (UID: \"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"
Sep 08 20:41:19.373297 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.373272    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-maas-api-tls\") pod \"maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl\" (UID: \"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"
Sep 08 20:41:19.373949 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.373926    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-metrics-certs\") pod \"maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl\" (UID: \"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"
Sep 08 20:41:19.378817 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.378793    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhtnm\" (UniqueName: \"kubernetes.io/projected/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-kube-api-access-xhtnm\") pod \"maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl\" (UID: \"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"
Sep 08 20:41:19.512285 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.512194    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"
Sep 08 20:41:19.807361 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.807279    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"]
Sep 08 20:41:19.813291 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.813271    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 20:41:19.864315 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.864279    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz"]
Sep 08 20:41:19.868686 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.868666    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz"
Sep 08 20:41:19.873669 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.873638    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-b433c5\""
Sep 08 20:41:19.885292 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.885271    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-b433c5-dockercfg-zlvw7\""
Sep 08 20:41:19.894647 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.894622    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz"]
Sep 08 20:41:19.976088 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.976020    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np97l\" (UniqueName: \"kubernetes.io/projected/066c84d8-c7aa-4d19-a9ef-c13e22a8e990-kube-api-access-np97l\") pod \"payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz\" (UID: \"066c84d8-c7aa-4d19-a9ef-c13e22a8e990\") " pod="openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz"
Sep 08 20:41:19.976364 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:19.976189    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/066c84d8-c7aa-4d19-a9ef-c13e22a8e990-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz\" (UID: \"066c84d8-c7aa-4d19-a9ef-c13e22a8e990\") " pod="openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz"
Sep 08 20:41:20.077356 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:20.077260    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/066c84d8-c7aa-4d19-a9ef-c13e22a8e990-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz\" (UID: \"066c84d8-c7aa-4d19-a9ef-c13e22a8e990\") " pod="openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz"
Sep 08 20:41:20.077356 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:20.077311    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np97l\" (UniqueName: \"kubernetes.io/projected/066c84d8-c7aa-4d19-a9ef-c13e22a8e990-kube-api-access-np97l\") pod \"payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz\" (UID: \"066c84d8-c7aa-4d19-a9ef-c13e22a8e990\") " pod="openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz"
Sep 08 20:41:20.077964 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:20.077942    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/066c84d8-c7aa-4d19-a9ef-c13e22a8e990-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz\" (UID: \"066c84d8-c7aa-4d19-a9ef-c13e22a8e990\") " pod="openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz"
Sep 08 20:41:20.084490 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:20.084461    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl" event={"ID":"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f","Type":"ContainerStarted","Data":"fbe4b850443c6ba2790c7f9ba0528583785a38ff1219bf516901a9e720a7a5dc"}
Sep 08 20:41:20.085786 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:20.085768    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np97l\" (UniqueName: \"kubernetes.io/projected/066c84d8-c7aa-4d19-a9ef-c13e22a8e990-kube-api-access-np97l\") pod \"payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz\" (UID: \"066c84d8-c7aa-4d19-a9ef-c13e22a8e990\") " pod="openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz"
Sep 08 20:41:20.178332 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:20.178309    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz"
Sep 08 20:41:20.450899 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:20.450875    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz"]
Sep 08 20:41:20.453334 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:41:20.453305    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066c84d8_c7aa_4d19_a9ef_c13e22a8e990.slice/crio-707191872e691c581408508559872c59b0f2389269cee8c5a0da8dba2e562f03 WatchSource:0}: Error finding container 707191872e691c581408508559872c59b0f2389269cee8c5a0da8dba2e562f03: Status 404 returned error can't find the container with id 707191872e691c581408508559872c59b0f2389269cee8c5a0da8dba2e562f03
Sep 08 20:41:21.091308 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:21.091269    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz" event={"ID":"066c84d8-c7aa-4d19-a9ef-c13e22a8e990","Type":"ContainerStarted","Data":"c96ecdd46736cfddbb55205effc3aea778de6dbc84e76d769f05cc7c73488fd0"}
Sep 08 20:41:21.091519 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:21.091331    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz" event={"ID":"066c84d8-c7aa-4d19-a9ef-c13e22a8e990","Type":"ContainerStarted","Data":"707191872e691c581408508559872c59b0f2389269cee8c5a0da8dba2e562f03"}
Sep 08 20:41:21.091519 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:21.091446    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz"
Sep 08 20:41:21.092817 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:21.092792    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl" event={"ID":"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f","Type":"ContainerStarted","Data":"606bff5f31491b7d340635b13d3114c66b317d38f21a482e57ed732663d3aa56"}
Sep 08 20:41:21.092921 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:21.092845    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"
Sep 08 20:41:21.108938 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:21.108888    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz" podStartSLOduration=2.108871894 podStartE2EDuration="2.108871894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:41:21.107471368 +0000 UTC m=+934.358952727" watchObservedRunningTime="2026-09-08 20:41:21.108871894 +0000 UTC m=+934.360353253"
Sep 08 20:41:21.125535 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:21.125490    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl" podStartSLOduration=1.807510357 podStartE2EDuration="2.12547704s" totalImagesPullingTime="317.966683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:41:21.123671215 +0000 UTC m=+934.375152574" watchObservedRunningTime="2026-09-08 20:41:21.12547704 +0000 UTC m=+934.376958396"
Sep 08 20:41:26.997475 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:26.997444    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:41:27.103523 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:27.103491    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"
Sep 08 20:41:32.100192 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:32.100156    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz"
Sep 08 20:41:43.788544 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:41:43.788506    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:42:04.200559 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:04.200527    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:42:18.718046 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:18.718012    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"]
Sep 08 20:42:18.718568 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:18.718211    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl" podUID="2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f" containerName="maas-api" containerID="cri-o://606bff5f31491b7d340635b13d3114c66b317d38f21a482e57ed732663d3aa56" gracePeriod=30
Sep 08 20:42:18.777812 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:18.777777    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz"]
Sep 08 20:42:18.778083 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:18.778039    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz" podUID="066c84d8-c7aa-4d19-a9ef-c13e22a8e990" containerName="payload-processing" containerID="cri-o://c96ecdd46736cfddbb55205effc3aea778de6dbc84e76d769f05cc7c73488fd0" gracePeriod=30
Sep 08 20:42:19.046563 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.046542    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"
Sep 08 20:42:19.049228 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.049212    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz"
Sep 08 20:42:19.181632 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.181594    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-openshift-service-ca\") pod \"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f\" (UID: \"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f\") "
Sep 08 20:42:19.181819 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.181658    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-maas-api-tls\") pod \"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f\" (UID: \"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f\") "
Sep 08 20:42:19.181819 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.181696    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhtnm\" (UniqueName: \"kubernetes.io/projected/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-kube-api-access-xhtnm\") pod \"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f\" (UID: \"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f\") "
Sep 08 20:42:19.181819 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.181719    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-np97l\" (UniqueName: \"kubernetes.io/projected/066c84d8-c7aa-4d19-a9ef-c13e22a8e990-kube-api-access-np97l\") pod \"066c84d8-c7aa-4d19-a9ef-c13e22a8e990\" (UID: \"066c84d8-c7aa-4d19-a9ef-c13e22a8e990\") "
Sep 08 20:42:19.181819 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.181809    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/066c84d8-c7aa-4d19-a9ef-c13e22a8e990-plugins-config-volume\") pod \"066c84d8-c7aa-4d19-a9ef-c13e22a8e990\" (UID: \"066c84d8-c7aa-4d19-a9ef-c13e22a8e990\") "
Sep 08 20:42:19.182009 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.181941    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-metrics-certs\") pod \"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f\" (UID: \"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f\") "
Sep 08 20:42:19.182107 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.182077    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f" (UID: "2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:42:19.182280 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.182262    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-openshift-service-ca\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:42:19.182361 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.182307    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/066c84d8-c7aa-4d19-a9ef-c13e22a8e990-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "066c84d8-c7aa-4d19-a9ef-c13e22a8e990" (UID: "066c84d8-c7aa-4d19-a9ef-c13e22a8e990"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:42:19.184178 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.184151    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f" (UID: "2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 20:42:19.184318 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.184297    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f" (UID: "2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 20:42:19.184392 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.184315    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-kube-api-access-xhtnm" (OuterVolumeSpecName: "kube-api-access-xhtnm") pod "2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f" (UID: "2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f"). InnerVolumeSpecName "kube-api-access-xhtnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:42:19.184392 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.184338    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/066c84d8-c7aa-4d19-a9ef-c13e22a8e990-kube-api-access-np97l" (OuterVolumeSpecName: "kube-api-access-np97l") pod "066c84d8-c7aa-4d19-a9ef-c13e22a8e990" (UID: "066c84d8-c7aa-4d19-a9ef-c13e22a8e990"). InnerVolumeSpecName "kube-api-access-np97l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:42:19.283297 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.283219    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-maas-api-tls\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:42:19.283297 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.283269    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhtnm\" (UniqueName: \"kubernetes.io/projected/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-kube-api-access-xhtnm\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:42:19.283297 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.283284    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np97l\" (UniqueName: \"kubernetes.io/projected/066c84d8-c7aa-4d19-a9ef-c13e22a8e990-kube-api-access-np97l\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:42:19.283297 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.283297    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/066c84d8-c7aa-4d19-a9ef-c13e22a8e990-plugins-config-volume\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:42:19.283505 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.283310    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f-metrics-certs\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:42:19.354802 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.354770    2552 generic.go:356] "Generic (PLEG): container finished" podID="2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f" containerID="606bff5f31491b7d340635b13d3114c66b317d38f21a482e57ed732663d3aa56" exitCode=0
Sep 08 20:42:19.354975 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.354814    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"
Sep 08 20:42:19.354975 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.354853    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl" event={"ID":"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f","Type":"ContainerDied","Data":"606bff5f31491b7d340635b13d3114c66b317d38f21a482e57ed732663d3aa56"}
Sep 08 20:42:19.354975 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.354903    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl" event={"ID":"2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f","Type":"ContainerDied","Data":"fbe4b850443c6ba2790c7f9ba0528583785a38ff1219bf516901a9e720a7a5dc"}
Sep 08 20:42:19.354975 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.354928    2552 scope.go:117] "RemoveContainer" containerID="606bff5f31491b7d340635b13d3114c66b317d38f21a482e57ed732663d3aa56"
Sep 08 20:42:19.356522 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.356383    2552 generic.go:356] "Generic (PLEG): container finished" podID="066c84d8-c7aa-4d19-a9ef-c13e22a8e990" containerID="c96ecdd46736cfddbb55205effc3aea778de6dbc84e76d769f05cc7c73488fd0" exitCode=0
Sep 08 20:42:19.356522 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.356428    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz" event={"ID":"066c84d8-c7aa-4d19-a9ef-c13e22a8e990","Type":"ContainerDied","Data":"c96ecdd46736cfddbb55205effc3aea778de6dbc84e76d769f05cc7c73488fd0"}
Sep 08 20:42:19.356522 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.356486    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz" event={"ID":"066c84d8-c7aa-4d19-a9ef-c13e22a8e990","Type":"ContainerDied","Data":"707191872e691c581408508559872c59b0f2389269cee8c5a0da8dba2e562f03"}
Sep 08 20:42:19.356522 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.356441    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz"
Sep 08 20:42:19.363987 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.363971    2552 scope.go:117] "RemoveContainer" containerID="606bff5f31491b7d340635b13d3114c66b317d38f21a482e57ed732663d3aa56"
Sep 08 20:42:19.364362 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:42:19.364341    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"606bff5f31491b7d340635b13d3114c66b317d38f21a482e57ed732663d3aa56\": container with ID starting with 606bff5f31491b7d340635b13d3114c66b317d38f21a482e57ed732663d3aa56 not found: ID does not exist" containerID="606bff5f31491b7d340635b13d3114c66b317d38f21a482e57ed732663d3aa56"
Sep 08 20:42:19.364426 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.364368    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606bff5f31491b7d340635b13d3114c66b317d38f21a482e57ed732663d3aa56"} err="failed to get container status \"606bff5f31491b7d340635b13d3114c66b317d38f21a482e57ed732663d3aa56\": rpc error: code = NotFound desc = could not find container \"606bff5f31491b7d340635b13d3114c66b317d38f21a482e57ed732663d3aa56\": container with ID starting with 606bff5f31491b7d340635b13d3114c66b317d38f21a482e57ed732663d3aa56 not found: ID does not exist"
Sep 08 20:42:19.364426 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.364386    2552 scope.go:117] "RemoveContainer" containerID="c96ecdd46736cfddbb55205effc3aea778de6dbc84e76d769f05cc7c73488fd0"
Sep 08 20:42:19.372136 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.372107    2552 scope.go:117] "RemoveContainer" containerID="c96ecdd46736cfddbb55205effc3aea778de6dbc84e76d769f05cc7c73488fd0"
Sep 08 20:42:19.372426 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:42:19.372405    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c96ecdd46736cfddbb55205effc3aea778de6dbc84e76d769f05cc7c73488fd0\": container with ID starting with c96ecdd46736cfddbb55205effc3aea778de6dbc84e76d769f05cc7c73488fd0 not found: ID does not exist" containerID="c96ecdd46736cfddbb55205effc3aea778de6dbc84e76d769f05cc7c73488fd0"
Sep 08 20:42:19.372486 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.372435    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c96ecdd46736cfddbb55205effc3aea778de6dbc84e76d769f05cc7c73488fd0"} err="failed to get container status \"c96ecdd46736cfddbb55205effc3aea778de6dbc84e76d769f05cc7c73488fd0\": rpc error: code = NotFound desc = could not find container \"c96ecdd46736cfddbb55205effc3aea778de6dbc84e76d769f05cc7c73488fd0\": container with ID starting with c96ecdd46736cfddbb55205effc3aea778de6dbc84e76d769f05cc7c73488fd0 not found: ID does not exist"
Sep 08 20:42:19.375401 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.375367    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz"]
Sep 08 20:42:19.378595 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.378575    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b433c5-8d6d87c-2zhwz"]
Sep 08 20:42:19.400507 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.400485    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"]
Sep 08 20:42:19.410604 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:19.410583    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b433c5-6d76568ccf-f9dtl"]
Sep 08 20:42:20.255257 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:20.255206    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:42:21.224001 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:21.223966    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="066c84d8-c7aa-4d19-a9ef-c13e22a8e990" path="/var/lib/kubelet/pods/066c84d8-c7aa-4d19-a9ef-c13e22a8e990/volumes"
Sep 08 20:42:21.224395 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:21.224381    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f" path="/var/lib/kubelet/pods/2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f/volumes"
Sep 08 20:42:31.700994 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:31.700955    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:42:38.108151 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.108095    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"]
Sep 08 20:42:38.108679 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.108634    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="066c84d8-c7aa-4d19-a9ef-c13e22a8e990" containerName="payload-processing"
Sep 08 20:42:38.108679 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.108655    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="066c84d8-c7aa-4d19-a9ef-c13e22a8e990" containerName="payload-processing"
Sep 08 20:42:38.108813 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.108688    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f" containerName="maas-api"
Sep 08 20:42:38.108813 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.108698    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f" containerName="maas-api"
Sep 08 20:42:38.108813 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.108775    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="066c84d8-c7aa-4d19-a9ef-c13e22a8e990" containerName="payload-processing"
Sep 08 20:42:38.108813 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.108790    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ba173ee-bb00-4a07-9c0c-f3f6304a8a2f" containerName="maas-api"
Sep 08 20:42:38.112155 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.112126    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"
Sep 08 20:42:38.114354 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.114324    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 20:42:38.114666 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.114647    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3e6b3c9b-serving-cert\""
Sep 08 20:42:38.115183 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.115166    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 20:42:38.115307 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.115166    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-z8244\""
Sep 08 20:42:38.115307 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.115230    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 20:42:38.125468 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.125442    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"]
Sep 08 20:42:38.151658 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.151628    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7373b9b3-9a43-449e-a970-86591d6528c0-openshift-service-ca\") pod \"maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb\" (UID: \"7373b9b3-9a43-449e-a970-86591d6528c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"
Sep 08 20:42:38.151658 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.151658    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbvrt\" (UniqueName: \"kubernetes.io/projected/7373b9b3-9a43-449e-a970-86591d6528c0-kube-api-access-jbvrt\") pod \"maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb\" (UID: \"7373b9b3-9a43-449e-a970-86591d6528c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"
Sep 08 20:42:38.151822 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.151690    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7373b9b3-9a43-449e-a970-86591d6528c0-maas-api-tls\") pod \"maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb\" (UID: \"7373b9b3-9a43-449e-a970-86591d6528c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"
Sep 08 20:42:38.151822 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.151790    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7373b9b3-9a43-449e-a970-86591d6528c0-metrics-certs\") pod \"maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb\" (UID: \"7373b9b3-9a43-449e-a970-86591d6528c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"
Sep 08 20:42:38.252483 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.252451    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7373b9b3-9a43-449e-a970-86591d6528c0-openshift-service-ca\") pod \"maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb\" (UID: \"7373b9b3-9a43-449e-a970-86591d6528c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"
Sep 08 20:42:38.252483 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.252484    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbvrt\" (UniqueName: \"kubernetes.io/projected/7373b9b3-9a43-449e-a970-86591d6528c0-kube-api-access-jbvrt\") pod \"maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb\" (UID: \"7373b9b3-9a43-449e-a970-86591d6528c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"
Sep 08 20:42:38.252734 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.252544    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7373b9b3-9a43-449e-a970-86591d6528c0-maas-api-tls\") pod \"maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb\" (UID: \"7373b9b3-9a43-449e-a970-86591d6528c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"
Sep 08 20:42:38.252734 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:42:38.252605    2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-serving-cert: secret "maas-api-e2e-mt-3e6b3c9b-serving-cert" not found
Sep 08 20:42:38.252734 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.252643    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7373b9b3-9a43-449e-a970-86591d6528c0-metrics-certs\") pod \"maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb\" (UID: \"7373b9b3-9a43-449e-a970-86591d6528c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"
Sep 08 20:42:38.252734 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:42:38.252677    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7373b9b3-9a43-449e-a970-86591d6528c0-maas-api-tls podName:7373b9b3-9a43-449e-a970-86591d6528c0 nodeName:}" failed. No retries permitted until 2026-09-08 20:42:38.752655938 +0000 UTC m=+1012.004137274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7373b9b3-9a43-449e-a970-86591d6528c0-maas-api-tls") pod "maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb" (UID: "7373b9b3-9a43-449e-a970-86591d6528c0") : secret "maas-api-e2e-mt-3e6b3c9b-serving-cert" not found
Sep 08 20:42:38.253121 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.253102    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7373b9b3-9a43-449e-a970-86591d6528c0-openshift-service-ca\") pod \"maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb\" (UID: \"7373b9b3-9a43-449e-a970-86591d6528c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"
Sep 08 20:42:38.254963 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.254945    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7373b9b3-9a43-449e-a970-86591d6528c0-metrics-certs\") pod \"maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb\" (UID: \"7373b9b3-9a43-449e-a970-86591d6528c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"
Sep 08 20:42:38.260826 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.260800    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbvrt\" (UniqueName: \"kubernetes.io/projected/7373b9b3-9a43-449e-a970-86591d6528c0-kube-api-access-jbvrt\") pod \"maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb\" (UID: \"7373b9b3-9a43-449e-a970-86591d6528c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"
Sep 08 20:42:38.706125 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.706091    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv"]
Sep 08 20:42:38.709346 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.709326    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv"
Sep 08 20:42:38.711801 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.711776    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3e6b3c9b-dockercfg-457ns\""
Sep 08 20:42:38.711904 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.711807    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3e6b3c9b\""
Sep 08 20:42:38.721428 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.721405    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv"]
Sep 08 20:42:38.757741 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.757706    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7373b9b3-9a43-449e-a970-86591d6528c0-maas-api-tls\") pod \"maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb\" (UID: \"7373b9b3-9a43-449e-a970-86591d6528c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"
Sep 08 20:42:38.757741 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.757741    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpgkr\" (UniqueName: \"kubernetes.io/projected/791b5184-4b2d-4a3c-83c9-9365ee71d02a-kube-api-access-gpgkr\") pod \"payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv\" (UID: \"791b5184-4b2d-4a3c-83c9-9365ee71d02a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv"
Sep 08 20:42:38.757982 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.757782    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/791b5184-4b2d-4a3c-83c9-9365ee71d02a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv\" (UID: \"791b5184-4b2d-4a3c-83c9-9365ee71d02a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv"
Sep 08 20:42:38.760127 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.760106    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7373b9b3-9a43-449e-a970-86591d6528c0-maas-api-tls\") pod \"maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb\" (UID: \"7373b9b3-9a43-449e-a970-86591d6528c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"
Sep 08 20:42:38.859139 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.859098    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/791b5184-4b2d-4a3c-83c9-9365ee71d02a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv\" (UID: \"791b5184-4b2d-4a3c-83c9-9365ee71d02a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv"
Sep 08 20:42:38.859307 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.859202    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpgkr\" (UniqueName: \"kubernetes.io/projected/791b5184-4b2d-4a3c-83c9-9365ee71d02a-kube-api-access-gpgkr\") pod \"payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv\" (UID: \"791b5184-4b2d-4a3c-83c9-9365ee71d02a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv"
Sep 08 20:42:38.859821 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.859802    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/791b5184-4b2d-4a3c-83c9-9365ee71d02a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv\" (UID: \"791b5184-4b2d-4a3c-83c9-9365ee71d02a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv"
Sep 08 20:42:38.872415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:38.872386    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpgkr\" (UniqueName: \"kubernetes.io/projected/791b5184-4b2d-4a3c-83c9-9365ee71d02a-kube-api-access-gpgkr\") pod \"payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv\" (UID: \"791b5184-4b2d-4a3c-83c9-9365ee71d02a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv"
Sep 08 20:42:39.020059 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:39.019973    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv"
Sep 08 20:42:39.022726 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:39.022703    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"
Sep 08 20:42:39.300341 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:39.300262    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"]
Sep 08 20:42:39.303549 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:39.303525    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv"]
Sep 08 20:42:39.303999 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:42:39.303972    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7373b9b3_9a43_449e_a970_86591d6528c0.slice/crio-f6236c5a394b53d3a64f13ca3f87c17d3c643e32224f4defe974077d955ff1a9 WatchSource:0}: Error finding container f6236c5a394b53d3a64f13ca3f87c17d3c643e32224f4defe974077d955ff1a9: Status 404 returned error can't find the container with id f6236c5a394b53d3a64f13ca3f87c17d3c643e32224f4defe974077d955ff1a9
Sep 08 20:42:39.305118 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:42:39.305093    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791b5184_4b2d_4a3c_83c9_9365ee71d02a.slice/crio-2b5fe153a59594182deb8d0e02880f3a364809354609c6fa3b9af64e9ec3696c WatchSource:0}: Error finding container 2b5fe153a59594182deb8d0e02880f3a364809354609c6fa3b9af64e9ec3696c: Status 404 returned error can't find the container with id 2b5fe153a59594182deb8d0e02880f3a364809354609c6fa3b9af64e9ec3696c
Sep 08 20:42:39.447504 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:39.447464    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv" event={"ID":"791b5184-4b2d-4a3c-83c9-9365ee71d02a","Type":"ContainerStarted","Data":"ca829012474c5d7228df440a97d9b41ed3ccf0731cba47b6e8f240f5a3c7b4ee"}
Sep 08 20:42:39.447695 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:39.447532    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv" event={"ID":"791b5184-4b2d-4a3c-83c9-9365ee71d02a","Type":"ContainerStarted","Data":"2b5fe153a59594182deb8d0e02880f3a364809354609c6fa3b9af64e9ec3696c"}
Sep 08 20:42:39.447695 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:39.447563    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv"
Sep 08 20:42:39.448949 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:39.448923    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb" event={"ID":"7373b9b3-9a43-449e-a970-86591d6528c0","Type":"ContainerStarted","Data":"f6236c5a394b53d3a64f13ca3f87c17d3c643e32224f4defe974077d955ff1a9"}
Sep 08 20:42:39.464587 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:39.464543    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv" podStartSLOduration=1.4645290389999999 podStartE2EDuration="1.464529039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:42:39.463209948 +0000 UTC m=+1012.714691305" watchObservedRunningTime="2026-09-08 20:42:39.464529039 +0000 UTC m=+1012.716010395"
Sep 08 20:42:40.461439 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:40.461396    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb" event={"ID":"7373b9b3-9a43-449e-a970-86591d6528c0","Type":"ContainerStarted","Data":"3952843c0b4e8018155c8e50a838c47ff422b93d14cc159f4f76db5c3b826167"}
Sep 08 20:42:40.461954 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:40.461770    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"
Sep 08 20:42:40.478185 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:40.478137    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb" podStartSLOduration=1.9641051109999998 podStartE2EDuration="2.4781204s" totalImagesPullingTime="514.015289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:42:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:42:40.476986704 +0000 UTC m=+1013.728468051" watchObservedRunningTime="2026-09-08 20:42:40.4781204 +0000 UTC m=+1013.729601760"
Sep 08 20:42:46.471823 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:46.471794    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"
Sep 08 20:42:47.003407 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:47.003369    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:42:50.463674 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:42:50.463645    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv"
Sep 08 20:43:06.899461 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:06.899426    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:43:18.195897 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:18.195863    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:43:19.564089 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:19.564055    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"]
Sep 08 20:43:19.564589 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:19.564323    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb" podUID="7373b9b3-9a43-449e-a970-86591d6528c0" containerName="maas-api" containerID="cri-o://3952843c0b4e8018155c8e50a838c47ff422b93d14cc159f4f76db5c3b826167" gracePeriod=30
Sep 08 20:43:19.626753 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:19.626716    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv"]
Sep 08 20:43:19.627138 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:19.626979    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv" podUID="791b5184-4b2d-4a3c-83c9-9365ee71d02a" containerName="payload-pre-processing" containerID="cri-o://ca829012474c5d7228df440a97d9b41ed3ccf0731cba47b6e8f240f5a3c7b4ee" gracePeriod=30
Sep 08 20:43:19.885560 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:19.885535    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"
Sep 08 20:43:19.888700 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:19.888677    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv"
Sep 08 20:43:19.915218 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:19.915192    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/791b5184-4b2d-4a3c-83c9-9365ee71d02a-plugins-config-volume\") pod \"791b5184-4b2d-4a3c-83c9-9365ee71d02a\" (UID: \"791b5184-4b2d-4a3c-83c9-9365ee71d02a\") "
Sep 08 20:43:19.915353 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:19.915286    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7373b9b3-9a43-449e-a970-86591d6528c0-maas-api-tls\") pod \"7373b9b3-9a43-449e-a970-86591d6528c0\" (UID: \"7373b9b3-9a43-449e-a970-86591d6528c0\") "
Sep 08 20:43:19.915353 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:19.915341    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpgkr\" (UniqueName: \"kubernetes.io/projected/791b5184-4b2d-4a3c-83c9-9365ee71d02a-kube-api-access-gpgkr\") pod \"791b5184-4b2d-4a3c-83c9-9365ee71d02a\" (UID: \"791b5184-4b2d-4a3c-83c9-9365ee71d02a\") "
Sep 08 20:43:19.915452 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:19.915373    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7373b9b3-9a43-449e-a970-86591d6528c0-metrics-certs\") pod \"7373b9b3-9a43-449e-a970-86591d6528c0\" (UID: \"7373b9b3-9a43-449e-a970-86591d6528c0\") "
Sep 08 20:43:19.915452 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:19.915426    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbvrt\" (UniqueName: \"kubernetes.io/projected/7373b9b3-9a43-449e-a970-86591d6528c0-kube-api-access-jbvrt\") pod \"7373b9b3-9a43-449e-a970-86591d6528c0\" (UID: \"7373b9b3-9a43-449e-a970-86591d6528c0\") "
Sep 08 20:43:19.915753 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:19.915621    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/791b5184-4b2d-4a3c-83c9-9365ee71d02a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "791b5184-4b2d-4a3c-83c9-9365ee71d02a" (UID: "791b5184-4b2d-4a3c-83c9-9365ee71d02a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:43:19.915753 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:19.915712    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7373b9b3-9a43-449e-a970-86591d6528c0-openshift-service-ca\") pod \"7373b9b3-9a43-449e-a970-86591d6528c0\" (UID: \"7373b9b3-9a43-449e-a970-86591d6528c0\") "
Sep 08 20:43:19.917516 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:19.916209    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7373b9b3-9a43-449e-a970-86591d6528c0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7373b9b3-9a43-449e-a970-86591d6528c0" (UID: "7373b9b3-9a43-449e-a970-86591d6528c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:43:19.917516 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:19.916677    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/791b5184-4b2d-4a3c-83c9-9365ee71d02a-plugins-config-volume\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:43:19.917516 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:19.916698    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7373b9b3-9a43-449e-a970-86591d6528c0-openshift-service-ca\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:43:19.917938 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:19.917897    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7373b9b3-9a43-449e-a970-86591d6528c0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7373b9b3-9a43-449e-a970-86591d6528c0" (UID: "7373b9b3-9a43-449e-a970-86591d6528c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 20:43:19.918043 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:19.918016    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/791b5184-4b2d-4a3c-83c9-9365ee71d02a-kube-api-access-gpgkr" (OuterVolumeSpecName: "kube-api-access-gpgkr") pod "791b5184-4b2d-4a3c-83c9-9365ee71d02a" (UID: "791b5184-4b2d-4a3c-83c9-9365ee71d02a"). InnerVolumeSpecName "kube-api-access-gpgkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:43:19.918218 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:19.918192    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7373b9b3-9a43-449e-a970-86591d6528c0-kube-api-access-jbvrt" (OuterVolumeSpecName: "kube-api-access-jbvrt") pod "7373b9b3-9a43-449e-a970-86591d6528c0" (UID: "7373b9b3-9a43-449e-a970-86591d6528c0"). InnerVolumeSpecName "kube-api-access-jbvrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:43:19.918356 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:19.918336    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7373b9b3-9a43-449e-a970-86591d6528c0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7373b9b3-9a43-449e-a970-86591d6528c0" (UID: "7373b9b3-9a43-449e-a970-86591d6528c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 20:43:20.017207 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:20.017177    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbvrt\" (UniqueName: \"kubernetes.io/projected/7373b9b3-9a43-449e-a970-86591d6528c0-kube-api-access-jbvrt\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:43:20.017207 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:20.017204    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7373b9b3-9a43-449e-a970-86591d6528c0-maas-api-tls\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:43:20.017406 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:20.017214    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpgkr\" (UniqueName: \"kubernetes.io/projected/791b5184-4b2d-4a3c-83c9-9365ee71d02a-kube-api-access-gpgkr\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:43:20.017406 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:20.017224    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7373b9b3-9a43-449e-a970-86591d6528c0-metrics-certs\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:43:20.639562 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:20.639525    2552 generic.go:356] "Generic (PLEG): container finished" podID="7373b9b3-9a43-449e-a970-86591d6528c0" containerID="3952843c0b4e8018155c8e50a838c47ff422b93d14cc159f4f76db5c3b826167" exitCode=0
Sep 08 20:43:20.639988 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:20.639576    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"
Sep 08 20:43:20.639988 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:20.639615    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb" event={"ID":"7373b9b3-9a43-449e-a970-86591d6528c0","Type":"ContainerDied","Data":"3952843c0b4e8018155c8e50a838c47ff422b93d14cc159f4f76db5c3b826167"}
Sep 08 20:43:20.639988 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:20.639687    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb" event={"ID":"7373b9b3-9a43-449e-a970-86591d6528c0","Type":"ContainerDied","Data":"f6236c5a394b53d3a64f13ca3f87c17d3c643e32224f4defe974077d955ff1a9"}
Sep 08 20:43:20.639988 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:20.639725    2552 scope.go:117] "RemoveContainer" containerID="3952843c0b4e8018155c8e50a838c47ff422b93d14cc159f4f76db5c3b826167"
Sep 08 20:43:20.641487 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:20.641214    2552 generic.go:356] "Generic (PLEG): container finished" podID="791b5184-4b2d-4a3c-83c9-9365ee71d02a" containerID="ca829012474c5d7228df440a97d9b41ed3ccf0731cba47b6e8f240f5a3c7b4ee" exitCode=0
Sep 08 20:43:20.641487 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:20.641267    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv" event={"ID":"791b5184-4b2d-4a3c-83c9-9365ee71d02a","Type":"ContainerDied","Data":"ca829012474c5d7228df440a97d9b41ed3ccf0731cba47b6e8f240f5a3c7b4ee"}
Sep 08 20:43:20.641487 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:20.641295    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv"
Sep 08 20:43:20.641487 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:20.641308    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv" event={"ID":"791b5184-4b2d-4a3c-83c9-9365ee71d02a","Type":"ContainerDied","Data":"2b5fe153a59594182deb8d0e02880f3a364809354609c6fa3b9af64e9ec3696c"}
Sep 08 20:43:20.649575 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:20.649560    2552 scope.go:117] "RemoveContainer" containerID="3952843c0b4e8018155c8e50a838c47ff422b93d14cc159f4f76db5c3b826167"
Sep 08 20:43:20.649852 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:43:20.649831    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3952843c0b4e8018155c8e50a838c47ff422b93d14cc159f4f76db5c3b826167\": container with ID starting with 3952843c0b4e8018155c8e50a838c47ff422b93d14cc159f4f76db5c3b826167 not found: ID does not exist" containerID="3952843c0b4e8018155c8e50a838c47ff422b93d14cc159f4f76db5c3b826167"
Sep 08 20:43:20.649952 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:20.649861    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3952843c0b4e8018155c8e50a838c47ff422b93d14cc159f4f76db5c3b826167"} err="failed to get container status \"3952843c0b4e8018155c8e50a838c47ff422b93d14cc159f4f76db5c3b826167\": rpc error: code = NotFound desc = could not find container \"3952843c0b4e8018155c8e50a838c47ff422b93d14cc159f4f76db5c3b826167\": container with ID starting with 3952843c0b4e8018155c8e50a838c47ff422b93d14cc159f4f76db5c3b826167 not found: ID does not exist"
Sep 08 20:43:20.649952 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:20.649876    2552 scope.go:117] "RemoveContainer" containerID="ca829012474c5d7228df440a97d9b41ed3ccf0731cba47b6e8f240f5a3c7b4ee"
Sep 08 20:43:20.657107 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:20.657089    2552 scope.go:117] "RemoveContainer" containerID="ca829012474c5d7228df440a97d9b41ed3ccf0731cba47b6e8f240f5a3c7b4ee"
Sep 08 20:43:20.657948 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:43:20.657932    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca829012474c5d7228df440a97d9b41ed3ccf0731cba47b6e8f240f5a3c7b4ee\": container with ID starting with ca829012474c5d7228df440a97d9b41ed3ccf0731cba47b6e8f240f5a3c7b4ee not found: ID does not exist" containerID="ca829012474c5d7228df440a97d9b41ed3ccf0731cba47b6e8f240f5a3c7b4ee"
Sep 08 20:43:20.658011 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:20.657954    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca829012474c5d7228df440a97d9b41ed3ccf0731cba47b6e8f240f5a3c7b4ee"} err="failed to get container status \"ca829012474c5d7228df440a97d9b41ed3ccf0731cba47b6e8f240f5a3c7b4ee\": rpc error: code = NotFound desc = could not find container \"ca829012474c5d7228df440a97d9b41ed3ccf0731cba47b6e8f240f5a3c7b4ee\": container with ID starting with ca829012474c5d7228df440a97d9b41ed3ccf0731cba47b6e8f240f5a3c7b4ee not found: ID does not exist"
Sep 08 20:43:20.661423 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:20.661395    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"]
Sep 08 20:43:20.663434 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:20.663415    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3e6b3c9b-6c65687cc7-bp7lb"]
Sep 08 20:43:20.673346 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:20.673326    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv"]
Sep 08 20:43:20.680158 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:20.680134    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3e6b3c9b-6df6764cbb-b6gwv"]
Sep 08 20:43:21.224345 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:21.224304    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7373b9b3-9a43-449e-a970-86591d6528c0" path="/var/lib/kubelet/pods/7373b9b3-9a43-449e-a970-86591d6528c0/volumes"
Sep 08 20:43:21.224747 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:21.224726    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="791b5184-4b2d-4a3c-83c9-9365ee71d02a" path="/var/lib/kubelet/pods/791b5184-4b2d-4a3c-83c9-9365ee71d02a/volumes"
Sep 08 20:43:27.196483 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:27.196445    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:43:34.712981 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:34.712943    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx"]
Sep 08 20:43:34.713475 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:34.713215    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx" podUID="04f7f517-cbc8-4e81-9155-a78ceec8fb06" containerName="payload-processing" containerID="cri-o://ffbb056666d91baacfb2768b7bb9252106e64f7c9cf411ab19c784ebc7e00d7f" gracePeriod=30
Sep 08 20:43:34.715302 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:34.715230    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2"]
Sep 08 20:43:34.715701 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:34.715650    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2" podUID="824b6925-412a-4bd6-b80a-288a8c27716d" containerName="payload-pre-processing" containerID="cri-o://9c335d5b74aee4c38729e9bdd5d489adcea73677f116af5cbc1dbda16a32335d" gracePeriod=30
Sep 08 20:43:34.989739 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:34.989714    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2"
Sep 08 20:43:34.992458 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:34.992440    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx"
Sep 08 20:43:35.048424 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.048396    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04f7f517-cbc8-4e81-9155-a78ceec8fb06-plugins-config-volume\") pod \"04f7f517-cbc8-4e81-9155-a78ceec8fb06\" (UID: \"04f7f517-cbc8-4e81-9155-a78ceec8fb06\") "
Sep 08 20:43:35.048593 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.048446    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgqb4\" (UniqueName: \"kubernetes.io/projected/04f7f517-cbc8-4e81-9155-a78ceec8fb06-kube-api-access-tgqb4\") pod \"04f7f517-cbc8-4e81-9155-a78ceec8fb06\" (UID: \"04f7f517-cbc8-4e81-9155-a78ceec8fb06\") "
Sep 08 20:43:35.048593 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.048499    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzd2k\" (UniqueName: \"kubernetes.io/projected/824b6925-412a-4bd6-b80a-288a8c27716d-kube-api-access-gzd2k\") pod \"824b6925-412a-4bd6-b80a-288a8c27716d\" (UID: \"824b6925-412a-4bd6-b80a-288a8c27716d\") "
Sep 08 20:43:35.048593 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.048554    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824b6925-412a-4bd6-b80a-288a8c27716d-plugins-config-volume\") pod \"824b6925-412a-4bd6-b80a-288a8c27716d\" (UID: \"824b6925-412a-4bd6-b80a-288a8c27716d\") "
Sep 08 20:43:35.048745 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.048713    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04f7f517-cbc8-4e81-9155-a78ceec8fb06-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "04f7f517-cbc8-4e81-9155-a78ceec8fb06" (UID: "04f7f517-cbc8-4e81-9155-a78ceec8fb06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:43:35.049006 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.048964    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/824b6925-412a-4bd6-b80a-288a8c27716d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "824b6925-412a-4bd6-b80a-288a8c27716d" (UID: "824b6925-412a-4bd6-b80a-288a8c27716d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:43:35.049006 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.048981    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04f7f517-cbc8-4e81-9155-a78ceec8fb06-plugins-config-volume\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:43:35.050616 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.050589    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824b6925-412a-4bd6-b80a-288a8c27716d-kube-api-access-gzd2k" (OuterVolumeSpecName: "kube-api-access-gzd2k") pod "824b6925-412a-4bd6-b80a-288a8c27716d" (UID: "824b6925-412a-4bd6-b80a-288a8c27716d"). InnerVolumeSpecName "kube-api-access-gzd2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:43:35.050718 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.050648    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f7f517-cbc8-4e81-9155-a78ceec8fb06-kube-api-access-tgqb4" (OuterVolumeSpecName: "kube-api-access-tgqb4") pod "04f7f517-cbc8-4e81-9155-a78ceec8fb06" (UID: "04f7f517-cbc8-4e81-9155-a78ceec8fb06"). InnerVolumeSpecName "kube-api-access-tgqb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:43:35.149683 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.149648    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824b6925-412a-4bd6-b80a-288a8c27716d-plugins-config-volume\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:43:35.149683 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.149676    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgqb4\" (UniqueName: \"kubernetes.io/projected/04f7f517-cbc8-4e81-9155-a78ceec8fb06-kube-api-access-tgqb4\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:43:35.149683 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.149686    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzd2k\" (UniqueName: \"kubernetes.io/projected/824b6925-412a-4bd6-b80a-288a8c27716d-kube-api-access-gzd2k\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:43:35.706760 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.706723    2552 generic.go:356] "Generic (PLEG): container finished" podID="824b6925-412a-4bd6-b80a-288a8c27716d" containerID="9c335d5b74aee4c38729e9bdd5d489adcea73677f116af5cbc1dbda16a32335d" exitCode=0
Sep 08 20:43:35.707049 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.706771    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2"
Sep 08 20:43:35.707049 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.706802    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2" event={"ID":"824b6925-412a-4bd6-b80a-288a8c27716d","Type":"ContainerDied","Data":"9c335d5b74aee4c38729e9bdd5d489adcea73677f116af5cbc1dbda16a32335d"}
Sep 08 20:43:35.707049 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.706856    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2" event={"ID":"824b6925-412a-4bd6-b80a-288a8c27716d","Type":"ContainerDied","Data":"0c3f152d0e4feb37774637002cbd1346d9bee9ead406fff2c7dd22b3f41e98eb"}
Sep 08 20:43:35.707049 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.706882    2552 scope.go:117] "RemoveContainer" containerID="9c335d5b74aee4c38729e9bdd5d489adcea73677f116af5cbc1dbda16a32335d"
Sep 08 20:43:35.708335 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.708311    2552 generic.go:356] "Generic (PLEG): container finished" podID="04f7f517-cbc8-4e81-9155-a78ceec8fb06" containerID="ffbb056666d91baacfb2768b7bb9252106e64f7c9cf411ab19c784ebc7e00d7f" exitCode=0
Sep 08 20:43:35.708432 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.708347    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx"
Sep 08 20:43:35.708432 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.708352    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx" event={"ID":"04f7f517-cbc8-4e81-9155-a78ceec8fb06","Type":"ContainerDied","Data":"ffbb056666d91baacfb2768b7bb9252106e64f7c9cf411ab19c784ebc7e00d7f"}
Sep 08 20:43:35.708432 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.708389    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx" event={"ID":"04f7f517-cbc8-4e81-9155-a78ceec8fb06","Type":"ContainerDied","Data":"43378e43b5c7d9d4c7d37b11c42aeb8a0b11786b3f549f84e355dd4e9d63512a"}
Sep 08 20:43:35.715448 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.715143    2552 scope.go:117] "RemoveContainer" containerID="9c335d5b74aee4c38729e9bdd5d489adcea73677f116af5cbc1dbda16a32335d"
Sep 08 20:43:35.715678 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:43:35.715451    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c335d5b74aee4c38729e9bdd5d489adcea73677f116af5cbc1dbda16a32335d\": container with ID starting with 9c335d5b74aee4c38729e9bdd5d489adcea73677f116af5cbc1dbda16a32335d not found: ID does not exist" containerID="9c335d5b74aee4c38729e9bdd5d489adcea73677f116af5cbc1dbda16a32335d"
Sep 08 20:43:35.715678 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.715481    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c335d5b74aee4c38729e9bdd5d489adcea73677f116af5cbc1dbda16a32335d"} err="failed to get container status \"9c335d5b74aee4c38729e9bdd5d489adcea73677f116af5cbc1dbda16a32335d\": rpc error: code = NotFound desc = could not find container \"9c335d5b74aee4c38729e9bdd5d489adcea73677f116af5cbc1dbda16a32335d\": container with ID starting with 9c335d5b74aee4c38729e9bdd5d489adcea73677f116af5cbc1dbda16a32335d not found: ID does not exist"
Sep 08 20:43:35.715678 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.715498    2552 scope.go:117] "RemoveContainer" containerID="ffbb056666d91baacfb2768b7bb9252106e64f7c9cf411ab19c784ebc7e00d7f"
Sep 08 20:43:35.723124 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.723103    2552 scope.go:117] "RemoveContainer" containerID="ffbb056666d91baacfb2768b7bb9252106e64f7c9cf411ab19c784ebc7e00d7f"
Sep 08 20:43:35.723442 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:43:35.723423    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffbb056666d91baacfb2768b7bb9252106e64f7c9cf411ab19c784ebc7e00d7f\": container with ID starting with ffbb056666d91baacfb2768b7bb9252106e64f7c9cf411ab19c784ebc7e00d7f not found: ID does not exist" containerID="ffbb056666d91baacfb2768b7bb9252106e64f7c9cf411ab19c784ebc7e00d7f"
Sep 08 20:43:35.723519 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.723454    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffbb056666d91baacfb2768b7bb9252106e64f7c9cf411ab19c784ebc7e00d7f"} err="failed to get container status \"ffbb056666d91baacfb2768b7bb9252106e64f7c9cf411ab19c784ebc7e00d7f\": rpc error: code = NotFound desc = could not find container \"ffbb056666d91baacfb2768b7bb9252106e64f7c9cf411ab19c784ebc7e00d7f\": container with ID starting with ffbb056666d91baacfb2768b7bb9252106e64f7c9cf411ab19c784ebc7e00d7f not found: ID does not exist"
Sep 08 20:43:35.723571 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.723537    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx"]
Sep 08 20:43:35.728111 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.728089    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a6137b-5f75ff9f-pf8tx"]
Sep 08 20:43:35.738118 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.738097    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2"]
Sep 08 20:43:35.744538 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:35.744512    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a6137b-cf47454b-7v9p2"]
Sep 08 20:43:37.225542 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:37.225512    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04f7f517-cbc8-4e81-9155-a78ceec8fb06" path="/var/lib/kubelet/pods/04f7f517-cbc8-4e81-9155-a78ceec8fb06/volumes"
Sep 08 20:43:37.225907 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:37.225848    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="824b6925-412a-4bd6-b80a-288a8c27716d" path="/var/lib/kubelet/pods/824b6925-412a-4bd6-b80a-288a8c27716d/volumes"
Sep 08 20:43:37.338664 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:37.338629    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:43:44.509365 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.509176    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"]
Sep 08 20:43:44.509766 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.509663    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791b5184-4b2d-4a3c-83c9-9365ee71d02a" containerName="payload-pre-processing"
Sep 08 20:43:44.509766 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.509682    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="791b5184-4b2d-4a3c-83c9-9365ee71d02a" containerName="payload-pre-processing"
Sep 08 20:43:44.509766 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.509693    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7373b9b3-9a43-449e-a970-86591d6528c0" containerName="maas-api"
Sep 08 20:43:44.509766 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.509702    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="7373b9b3-9a43-449e-a970-86591d6528c0" containerName="maas-api"
Sep 08 20:43:44.509766 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.509718    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04f7f517-cbc8-4e81-9155-a78ceec8fb06" containerName="payload-processing"
Sep 08 20:43:44.509766 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.509727    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="04f7f517-cbc8-4e81-9155-a78ceec8fb06" containerName="payload-processing"
Sep 08 20:43:44.509766 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.509758    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="824b6925-412a-4bd6-b80a-288a8c27716d" containerName="payload-pre-processing"
Sep 08 20:43:44.509766 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.509766    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="824b6925-412a-4bd6-b80a-288a8c27716d" containerName="payload-pre-processing"
Sep 08 20:43:44.510184 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.509844    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="7373b9b3-9a43-449e-a970-86591d6528c0" containerName="maas-api"
Sep 08 20:43:44.510184 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.509855    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="04f7f517-cbc8-4e81-9155-a78ceec8fb06" containerName="payload-processing"
Sep 08 20:43:44.510184 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.509866    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="824b6925-412a-4bd6-b80a-288a8c27716d" containerName="payload-pre-processing"
Sep 08 20:43:44.510184 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.509874    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="791b5184-4b2d-4a3c-83c9-9365ee71d02a" containerName="payload-pre-processing"
Sep 08 20:43:44.514673 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.514652    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"
Sep 08 20:43:44.517408 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.517384    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-27d4c1-serving-cert\""
Sep 08 20:43:44.517540 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.517463    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 20:43:44.518456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.518435    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-z8244\""
Sep 08 20:43:44.518565 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.518474    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 20:43:44.518565 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.518496    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 20:43:44.523986 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.523962    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"]
Sep 08 20:43:44.637323 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.637283    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmtsr\" (UniqueName: \"kubernetes.io/projected/fd1d278e-bc11-43f3-88fa-195158176a34-kube-api-access-xmtsr\") pod \"maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw\" (UID: \"fd1d278e-bc11-43f3-88fa-195158176a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"
Sep 08 20:43:44.637508 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.637338    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1d278e-bc11-43f3-88fa-195158176a34-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw\" (UID: \"fd1d278e-bc11-43f3-88fa-195158176a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"
Sep 08 20:43:44.637508 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.637402    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1d278e-bc11-43f3-88fa-195158176a34-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw\" (UID: \"fd1d278e-bc11-43f3-88fa-195158176a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"
Sep 08 20:43:44.637508 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.637479    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1d278e-bc11-43f3-88fa-195158176a34-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw\" (UID: \"fd1d278e-bc11-43f3-88fa-195158176a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"
Sep 08 20:43:44.738497 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.738462    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmtsr\" (UniqueName: \"kubernetes.io/projected/fd1d278e-bc11-43f3-88fa-195158176a34-kube-api-access-xmtsr\") pod \"maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw\" (UID: \"fd1d278e-bc11-43f3-88fa-195158176a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"
Sep 08 20:43:44.738666 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.738511    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1d278e-bc11-43f3-88fa-195158176a34-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw\" (UID: \"fd1d278e-bc11-43f3-88fa-195158176a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"
Sep 08 20:43:44.738666 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.738543    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1d278e-bc11-43f3-88fa-195158176a34-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw\" (UID: \"fd1d278e-bc11-43f3-88fa-195158176a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"
Sep 08 20:43:44.738666 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.738585    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1d278e-bc11-43f3-88fa-195158176a34-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw\" (UID: \"fd1d278e-bc11-43f3-88fa-195158176a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"
Sep 08 20:43:44.739314 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.739293    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1d278e-bc11-43f3-88fa-195158176a34-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw\" (UID: \"fd1d278e-bc11-43f3-88fa-195158176a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"
Sep 08 20:43:44.741103 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.741075    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1d278e-bc11-43f3-88fa-195158176a34-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw\" (UID: \"fd1d278e-bc11-43f3-88fa-195158176a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"
Sep 08 20:43:44.741201 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.741154    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1d278e-bc11-43f3-88fa-195158176a34-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw\" (UID: \"fd1d278e-bc11-43f3-88fa-195158176a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"
Sep 08 20:43:44.746720 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.746693    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmtsr\" (UniqueName: \"kubernetes.io/projected/fd1d278e-bc11-43f3-88fa-195158176a34-kube-api-access-xmtsr\") pod \"maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw\" (UID: \"fd1d278e-bc11-43f3-88fa-195158176a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"
Sep 08 20:43:44.827681 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:44.827591    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"
Sep 08 20:43:45.098745 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:45.098719    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"]
Sep 08 20:43:45.101549 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:43:45.101523    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1d278e_bc11_43f3_88fa_195158176a34.slice/crio-cdb4da567ec9a30036cc6521d85034c0ff97d69bbae5c95d088d163b4d4fb0d7 WatchSource:0}: Error finding container cdb4da567ec9a30036cc6521d85034c0ff97d69bbae5c95d088d163b4d4fb0d7: Status 404 returned error can't find the container with id cdb4da567ec9a30036cc6521d85034c0ff97d69bbae5c95d088d163b4d4fb0d7
Sep 08 20:43:45.754115 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:45.754086    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw" event={"ID":"fd1d278e-bc11-43f3-88fa-195158176a34","Type":"ContainerStarted","Data":"43b7b9dc7cc92e0075d00c3ae9d0639bd6f9e44e954dee3617df1334125549bf"}
Sep 08 20:43:45.754504 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:45.754143    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw" event={"ID":"fd1d278e-bc11-43f3-88fa-195158176a34","Type":"ContainerStarted","Data":"cdb4da567ec9a30036cc6521d85034c0ff97d69bbae5c95d088d163b4d4fb0d7"}
Sep 08 20:43:45.754504 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:45.754230    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"
Sep 08 20:43:45.776418 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:45.776371    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw" podStartSLOduration=1.295437337 podStartE2EDuration="1.776354584s" totalImagesPullingTime="480.917247ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:43:45.77483686 +0000 UTC m=+1079.026318216" watchObservedRunningTime="2026-09-08 20:43:45.776354584 +0000 UTC m=+1079.027835937"
Sep 08 20:43:47.899438 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:47.899401    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd"]
Sep 08 20:43:47.904089 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:47.904071    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd"
Sep 08 20:43:47.906713 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:47.906688    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-78dd9f-dockercfg-cndcv\""
Sep 08 20:43:47.907019 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:47.906997    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-78dd9f\""
Sep 08 20:43:47.913411 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:47.913379    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd"]
Sep 08 20:43:48.065989 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:48.065948    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a663a5f-8573-4d0e-ae77-084ad099c501-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd\" (UID: \"5a663a5f-8573-4d0e-ae77-084ad099c501\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd"
Sep 08 20:43:48.065989 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:48.065988    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h82xf\" (UniqueName: \"kubernetes.io/projected/5a663a5f-8573-4d0e-ae77-084ad099c501-kube-api-access-h82xf\") pod \"payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd\" (UID: \"5a663a5f-8573-4d0e-ae77-084ad099c501\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd"
Sep 08 20:43:48.167453 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:48.167366    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a663a5f-8573-4d0e-ae77-084ad099c501-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd\" (UID: \"5a663a5f-8573-4d0e-ae77-084ad099c501\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd"
Sep 08 20:43:48.167453 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:48.167401    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h82xf\" (UniqueName: \"kubernetes.io/projected/5a663a5f-8573-4d0e-ae77-084ad099c501-kube-api-access-h82xf\") pod \"payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd\" (UID: \"5a663a5f-8573-4d0e-ae77-084ad099c501\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd"
Sep 08 20:43:48.168031 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:48.168012    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a663a5f-8573-4d0e-ae77-084ad099c501-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd\" (UID: \"5a663a5f-8573-4d0e-ae77-084ad099c501\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd"
Sep 08 20:43:48.175805 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:48.175774    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h82xf\" (UniqueName: \"kubernetes.io/projected/5a663a5f-8573-4d0e-ae77-084ad099c501-kube-api-access-h82xf\") pod \"payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd\" (UID: \"5a663a5f-8573-4d0e-ae77-084ad099c501\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd"
Sep 08 20:43:48.215373 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:48.215345    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd"
Sep 08 20:43:48.490689 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:48.490599    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd"]
Sep 08 20:43:48.492909 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:43:48.492880    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a663a5f_8573_4d0e_ae77_084ad099c501.slice/crio-32ced29ba77ff291469091fa2d6507006b208cb26202d4cde29fd274c3e2e17e WatchSource:0}: Error finding container 32ced29ba77ff291469091fa2d6507006b208cb26202d4cde29fd274c3e2e17e: Status 404 returned error can't find the container with id 32ced29ba77ff291469091fa2d6507006b208cb26202d4cde29fd274c3e2e17e
Sep 08 20:43:48.768750 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:48.768658    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd" event={"ID":"5a663a5f-8573-4d0e-ae77-084ad099c501","Type":"ContainerStarted","Data":"0699a044b7f7bcf4c4ec86cbdcd0759994625fca8feb53256da2d8200ee36ecf"}
Sep 08 20:43:48.768750 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:48.768728    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd" event={"ID":"5a663a5f-8573-4d0e-ae77-084ad099c501","Type":"ContainerStarted","Data":"32ced29ba77ff291469091fa2d6507006b208cb26202d4cde29fd274c3e2e17e"}
Sep 08 20:43:48.768954 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:48.768831    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd"
Sep 08 20:43:48.786294 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:48.786228    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd" podStartSLOduration=1.786214192 podStartE2EDuration="1.786214192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:43:48.785109687 +0000 UTC m=+1082.036591046" watchObservedRunningTime="2026-09-08 20:43:48.786214192 +0000 UTC m=+1082.037695577"
Sep 08 20:43:51.392820 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:51.392782    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:43:51.764386 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:51.764300    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"
Sep 08 20:43:59.775954 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:43:59.775918    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd"
Sep 08 20:44:13.381285 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:13.381227    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"]
Sep 08 20:44:13.384964 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:13.384940    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"
Sep 08 20:44:13.388332 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:13.388309    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-74f9a1-serving-cert\""
Sep 08 20:44:13.394871 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:13.394844    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"]
Sep 08 20:44:13.490177 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:13.490135    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf\" (UID: \"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"
Sep 08 20:44:13.490177 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:13.490175    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2mpt\" (UniqueName: \"kubernetes.io/projected/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-kube-api-access-p2mpt\") pod \"maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf\" (UID: \"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"
Sep 08 20:44:13.490414 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:13.490203    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf\" (UID: \"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"
Sep 08 20:44:13.490414 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:13.490288    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf\" (UID: \"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"
Sep 08 20:44:13.591216 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:13.591183    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf\" (UID: \"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"
Sep 08 20:44:13.591216 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:13.591224    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2mpt\" (UniqueName: \"kubernetes.io/projected/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-kube-api-access-p2mpt\") pod \"maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf\" (UID: \"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"
Sep 08 20:44:13.591451 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:13.591272    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf\" (UID: \"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"
Sep 08 20:44:13.591451 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:13.591289    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf\" (UID: \"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"
Sep 08 20:44:13.591529 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:44:13.591443    2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-serving-cert: secret "maas-api-e2e-shared-b-w2-74f9a1-serving-cert" not found
Sep 08 20:44:13.591564 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:44:13.591534    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-maas-api-tls podName:090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13 nodeName:}" failed. No retries permitted until 2026-09-08 20:44:14.09151249 +0000 UTC m=+1107.342993830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-maas-api-tls") pod "maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf" (UID: "090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13") : secret "maas-api-e2e-shared-b-w2-74f9a1-serving-cert" not found
Sep 08 20:44:13.591904 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:13.591880    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf\" (UID: \"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"
Sep 08 20:44:13.593635 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:13.593615    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf\" (UID: \"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"
Sep 08 20:44:13.601718 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:13.601695    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2mpt\" (UniqueName: \"kubernetes.io/projected/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-kube-api-access-p2mpt\") pod \"maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf\" (UID: \"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"
Sep 08 20:44:14.096096 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:14.096064    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf\" (UID: \"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"
Sep 08 20:44:14.098443 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:14.098413    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf\" (UID: \"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"
Sep 08 20:44:14.297727 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:14.297689    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"
Sep 08 20:44:14.571693 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:14.571669    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"]
Sep 08 20:44:14.573591 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:44:14.573561    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090e91e2_a6cf_4a3a_9cbf_ab1ea9710e13.slice/crio-ba46367695d44daf9caf280248d942fb7f3361ced512f23b2fdb17a6ae4bf574 WatchSource:0}: Error finding container ba46367695d44daf9caf280248d942fb7f3361ced512f23b2fdb17a6ae4bf574: Status 404 returned error can't find the container with id ba46367695d44daf9caf280248d942fb7f3361ced512f23b2fdb17a6ae4bf574
Sep 08 20:44:14.878581 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:14.878539    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf" event={"ID":"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13","Type":"ContainerStarted","Data":"ba46367695d44daf9caf280248d942fb7f3361ced512f23b2fdb17a6ae4bf574"}
Sep 08 20:44:15.885017 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:15.884986    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf" event={"ID":"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13","Type":"ContainerStarted","Data":"9a877a0e0182fc0da7eed75669ebc4aa756e9e39bf703690b49e42f122b1b31d"}
Sep 08 20:44:15.885416 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:15.885104    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"
Sep 08 20:44:15.902688 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:15.902635    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf" podStartSLOduration=2.07142179 podStartE2EDuration="2.902619125s" totalImagesPullingTime="831.197335ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:44:15.90112908 +0000 UTC m=+1109.152610441" watchObservedRunningTime="2026-09-08 20:44:15.902619125 +0000 UTC m=+1109.154100483"
Sep 08 20:44:16.269759 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:16.269678    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr"]
Sep 08 20:44:16.272991 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:16.272974    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr"
Sep 08 20:44:16.275623 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:16.275595    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-c9590b\""
Sep 08 20:44:16.275623 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:16.275622    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-c9590b-dockercfg-xm6l2\""
Sep 08 20:44:16.283596 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:16.283573    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr"]
Sep 08 20:44:16.315471 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:16.315435    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c505130-34b0-4a53-81b4-fd0c9efeb586-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr\" (UID: \"4c505130-34b0-4a53-81b4-fd0c9efeb586\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr"
Sep 08 20:44:16.315646 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:16.315510    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxdgk\" (UniqueName: \"kubernetes.io/projected/4c505130-34b0-4a53-81b4-fd0c9efeb586-kube-api-access-nxdgk\") pod \"payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr\" (UID: \"4c505130-34b0-4a53-81b4-fd0c9efeb586\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr"
Sep 08 20:44:16.416980 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:16.416941    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxdgk\" (UniqueName: \"kubernetes.io/projected/4c505130-34b0-4a53-81b4-fd0c9efeb586-kube-api-access-nxdgk\") pod \"payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr\" (UID: \"4c505130-34b0-4a53-81b4-fd0c9efeb586\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr"
Sep 08 20:44:16.417159 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:16.417032    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c505130-34b0-4a53-81b4-fd0c9efeb586-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr\" (UID: \"4c505130-34b0-4a53-81b4-fd0c9efeb586\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr"
Sep 08 20:44:16.417796 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:16.417776    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c505130-34b0-4a53-81b4-fd0c9efeb586-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr\" (UID: \"4c505130-34b0-4a53-81b4-fd0c9efeb586\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr"
Sep 08 20:44:16.426475 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:16.426448    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxdgk\" (UniqueName: \"kubernetes.io/projected/4c505130-34b0-4a53-81b4-fd0c9efeb586-kube-api-access-nxdgk\") pod \"payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr\" (UID: \"4c505130-34b0-4a53-81b4-fd0c9efeb586\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr"
Sep 08 20:44:16.584376 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:16.584341    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr"
Sep 08 20:44:16.856787 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:16.856713    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr"]
Sep 08 20:44:16.860427 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:44:16.860401    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c505130_34b0_4a53_81b4_fd0c9efeb586.slice/crio-5af4f3062806f4ad095c3ec7970b4bb0db60f4bbb79e16434432a4c61b45a5da WatchSource:0}: Error finding container 5af4f3062806f4ad095c3ec7970b4bb0db60f4bbb79e16434432a4c61b45a5da: Status 404 returned error can't find the container with id 5af4f3062806f4ad095c3ec7970b4bb0db60f4bbb79e16434432a4c61b45a5da
Sep 08 20:44:16.890469 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:16.890444    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr" event={"ID":"4c505130-34b0-4a53-81b4-fd0c9efeb586","Type":"ContainerStarted","Data":"5af4f3062806f4ad095c3ec7970b4bb0db60f4bbb79e16434432a4c61b45a5da"}
Sep 08 20:44:17.896630 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:17.896597    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr" event={"ID":"4c505130-34b0-4a53-81b4-fd0c9efeb586","Type":"ContainerStarted","Data":"c040e47fbaf753570db145b59a9dcc06e2e0d231dfad21e49a02733091cd640e"}
Sep 08 20:44:17.897048 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:17.896696    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr"
Sep 08 20:44:17.929889 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:17.929840    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr" podStartSLOduration=1.9298239339999999 podStartE2EDuration="1.929823934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:44:17.928722652 +0000 UTC m=+1111.180204011" watchObservedRunningTime="2026-09-08 20:44:17.929823934 +0000 UTC m=+1111.181305291"
Sep 08 20:44:20.794838 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:20.794807    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:44:21.895860 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:21.895826    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"
Sep 08 20:44:28.903605 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:28.903575    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr"
Sep 08 20:44:54.907762 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:54.907726    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"]
Sep 08 20:44:54.913106 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:54.913084    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"
Sep 08 20:44:54.916831 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:54.916808    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e0dc0027-serving-cert\""
Sep 08 20:44:54.917489 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:54.917468    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:44:54.924971 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:54.924951    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"]
Sep 08 20:44:54.955926 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:54.955895    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/357d86f6-d540-4d8a-824e-be2a4a7a6716-metrics-certs\") pod \"maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2\" (UID: \"357d86f6-d540-4d8a-824e-be2a4a7a6716\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"
Sep 08 20:44:54.956054 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:54.955941    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/357d86f6-d540-4d8a-824e-be2a4a7a6716-openshift-service-ca\") pod \"maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2\" (UID: \"357d86f6-d540-4d8a-824e-be2a4a7a6716\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"
Sep 08 20:44:54.956127 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:54.956070    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qmmr\" (UniqueName: \"kubernetes.io/projected/357d86f6-d540-4d8a-824e-be2a4a7a6716-kube-api-access-2qmmr\") pod \"maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2\" (UID: \"357d86f6-d540-4d8a-824e-be2a4a7a6716\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"
Sep 08 20:44:54.956127 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:54.956114    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/357d86f6-d540-4d8a-824e-be2a4a7a6716-maas-api-tls\") pod \"maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2\" (UID: \"357d86f6-d540-4d8a-824e-be2a4a7a6716\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"
Sep 08 20:44:55.057048 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:55.057009    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/357d86f6-d540-4d8a-824e-be2a4a7a6716-openshift-service-ca\") pod \"maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2\" (UID: \"357d86f6-d540-4d8a-824e-be2a4a7a6716\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"
Sep 08 20:44:55.057220 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:55.057054    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qmmr\" (UniqueName: \"kubernetes.io/projected/357d86f6-d540-4d8a-824e-be2a4a7a6716-kube-api-access-2qmmr\") pod \"maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2\" (UID: \"357d86f6-d540-4d8a-824e-be2a4a7a6716\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"
Sep 08 20:44:55.057220 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:55.057088    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/357d86f6-d540-4d8a-824e-be2a4a7a6716-maas-api-tls\") pod \"maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2\" (UID: \"357d86f6-d540-4d8a-824e-be2a4a7a6716\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"
Sep 08 20:44:55.057220 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:55.057210    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/357d86f6-d540-4d8a-824e-be2a4a7a6716-metrics-certs\") pod \"maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2\" (UID: \"357d86f6-d540-4d8a-824e-be2a4a7a6716\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"
Sep 08 20:44:55.057404 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:44:55.057331    2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-serving-cert: secret "maas-api-e2e-mt-e0dc0027-serving-cert" not found
Sep 08 20:44:55.057455 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:44:55.057406    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/357d86f6-d540-4d8a-824e-be2a4a7a6716-maas-api-tls podName:357d86f6-d540-4d8a-824e-be2a4a7a6716 nodeName:}" failed. No retries permitted until 2026-09-08 20:44:55.557385608 +0000 UTC m=+1148.808866948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/357d86f6-d540-4d8a-824e-be2a4a7a6716-maas-api-tls") pod "maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2" (UID: "357d86f6-d540-4d8a-824e-be2a4a7a6716") : secret "maas-api-e2e-mt-e0dc0027-serving-cert" not found
Sep 08 20:44:55.057759 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:55.057736    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/357d86f6-d540-4d8a-824e-be2a4a7a6716-openshift-service-ca\") pod \"maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2\" (UID: \"357d86f6-d540-4d8a-824e-be2a4a7a6716\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"
Sep 08 20:44:55.059621 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:55.059599    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/357d86f6-d540-4d8a-824e-be2a4a7a6716-metrics-certs\") pod \"maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2\" (UID: \"357d86f6-d540-4d8a-824e-be2a4a7a6716\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"
Sep 08 20:44:55.066403 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:55.066384    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qmmr\" (UniqueName: \"kubernetes.io/projected/357d86f6-d540-4d8a-824e-be2a4a7a6716-kube-api-access-2qmmr\") pod \"maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2\" (UID: \"357d86f6-d540-4d8a-824e-be2a4a7a6716\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"
Sep 08 20:44:55.560925 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:55.560890    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/357d86f6-d540-4d8a-824e-be2a4a7a6716-maas-api-tls\") pod \"maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2\" (UID: \"357d86f6-d540-4d8a-824e-be2a4a7a6716\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"
Sep 08 20:44:55.561083 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:44:55.561008    2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-serving-cert: secret "maas-api-e2e-mt-e0dc0027-serving-cert" not found
Sep 08 20:44:55.561083 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:44:55.561070    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/357d86f6-d540-4d8a-824e-be2a4a7a6716-maas-api-tls podName:357d86f6-d540-4d8a-824e-be2a4a7a6716 nodeName:}" failed. No retries permitted until 2026-09-08 20:44:56.561056001 +0000 UTC m=+1149.812537335 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/357d86f6-d540-4d8a-824e-be2a4a7a6716-maas-api-tls") pod "maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2" (UID: "357d86f6-d540-4d8a-824e-be2a4a7a6716") : secret "maas-api-e2e-mt-e0dc0027-serving-cert" not found
Sep 08 20:44:55.631660 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:55.631627    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59"]
Sep 08 20:44:55.635462 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:55.635441    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59"
Sep 08 20:44:55.637900 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:55.637878    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e0dc0027\""
Sep 08 20:44:55.638024 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:55.637937    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e0dc0027-dockercfg-w2lcg\""
Sep 08 20:44:55.650661 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:55.650639    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59"]
Sep 08 20:44:55.661807 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:55.661780    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nft6n\" (UniqueName: \"kubernetes.io/projected/8b11f91c-1280-46e6-ba91-faedc2bb61da-kube-api-access-nft6n\") pod \"payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59\" (UID: \"8b11f91c-1280-46e6-ba91-faedc2bb61da\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59"
Sep 08 20:44:55.661927 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:55.661908    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b11f91c-1280-46e6-ba91-faedc2bb61da-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59\" (UID: \"8b11f91c-1280-46e6-ba91-faedc2bb61da\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59"
Sep 08 20:44:55.762651 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:55.762619    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b11f91c-1280-46e6-ba91-faedc2bb61da-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59\" (UID: \"8b11f91c-1280-46e6-ba91-faedc2bb61da\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59"
Sep 08 20:44:55.762825 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:55.762681    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nft6n\" (UniqueName: \"kubernetes.io/projected/8b11f91c-1280-46e6-ba91-faedc2bb61da-kube-api-access-nft6n\") pod \"payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59\" (UID: \"8b11f91c-1280-46e6-ba91-faedc2bb61da\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59"
Sep 08 20:44:55.763375 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:55.763353    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b11f91c-1280-46e6-ba91-faedc2bb61da-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59\" (UID: \"8b11f91c-1280-46e6-ba91-faedc2bb61da\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59"
Sep 08 20:44:55.771346 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:55.771316    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nft6n\" (UniqueName: \"kubernetes.io/projected/8b11f91c-1280-46e6-ba91-faedc2bb61da-kube-api-access-nft6n\") pod \"payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59\" (UID: \"8b11f91c-1280-46e6-ba91-faedc2bb61da\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59"
Sep 08 20:44:55.946039 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:55.946008    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59"
Sep 08 20:44:56.218147 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:56.218116    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59"]
Sep 08 20:44:56.220140 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:44:56.220114    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b11f91c_1280_46e6_ba91_faedc2bb61da.slice/crio-c6f7782f19cea90ef76902a4d9be78403ec8703f4b4a21fc6c49b394b2096368 WatchSource:0}: Error finding container c6f7782f19cea90ef76902a4d9be78403ec8703f4b4a21fc6c49b394b2096368: Status 404 returned error can't find the container with id c6f7782f19cea90ef76902a4d9be78403ec8703f4b4a21fc6c49b394b2096368
Sep 08 20:44:56.569750 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:56.569654    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/357d86f6-d540-4d8a-824e-be2a4a7a6716-maas-api-tls\") pod \"maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2\" (UID: \"357d86f6-d540-4d8a-824e-be2a4a7a6716\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"
Sep 08 20:44:56.572046 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:56.572020    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/357d86f6-d540-4d8a-824e-be2a4a7a6716-maas-api-tls\") pod \"maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2\" (UID: \"357d86f6-d540-4d8a-824e-be2a4a7a6716\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"
Sep 08 20:44:56.724331 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:56.724293    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"
Sep 08 20:44:56.995616 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:56.995593    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"]
Sep 08 20:44:56.997433 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:44:56.997399    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357d86f6_d540_4d8a_824e_be2a4a7a6716.slice/crio-10d8d8d0c886e91321f3347922e0d9678e68f9409321e3feef9befe9cb8be536 WatchSource:0}: Error finding container 10d8d8d0c886e91321f3347922e0d9678e68f9409321e3feef9befe9cb8be536: Status 404 returned error can't find the container with id 10d8d8d0c886e91321f3347922e0d9678e68f9409321e3feef9befe9cb8be536
Sep 08 20:44:57.071807 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:57.071768    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59" event={"ID":"8b11f91c-1280-46e6-ba91-faedc2bb61da","Type":"ContainerStarted","Data":"9be2ef1c4d3feb27750202c5f6b9279d7be851d0de403a3ad75bc82da36f334d"}
Sep 08 20:44:57.071979 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:57.071924    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59"
Sep 08 20:44:57.071979 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:57.071960    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59" event={"ID":"8b11f91c-1280-46e6-ba91-faedc2bb61da","Type":"ContainerStarted","Data":"c6f7782f19cea90ef76902a4d9be78403ec8703f4b4a21fc6c49b394b2096368"}
Sep 08 20:44:57.073037 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:57.073011    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2" event={"ID":"357d86f6-d540-4d8a-824e-be2a4a7a6716","Type":"ContainerStarted","Data":"10d8d8d0c886e91321f3347922e0d9678e68f9409321e3feef9befe9cb8be536"}
Sep 08 20:44:57.086420 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:57.086359    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59" podStartSLOduration=2.086342969 podStartE2EDuration="2.086342969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:44:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:44:57.085731592 +0000 UTC m=+1150.337212950" watchObservedRunningTime="2026-09-08 20:44:57.086342969 +0000 UTC m=+1150.337824327"
Sep 08 20:44:58.078797 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:58.078711    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2" event={"ID":"357d86f6-d540-4d8a-824e-be2a4a7a6716","Type":"ContainerStarted","Data":"0aa27413dedbab51a6685f21f84958ee465927e05a182e7329d12cb549327885"}
Sep 08 20:44:58.078797 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:58.078791    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"
Sep 08 20:44:58.096606 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:44:58.096562    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2" podStartSLOduration=3.391188757 podStartE2EDuration="4.096547093s" totalImagesPullingTime="705.358336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:44:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:44:58.094435839 +0000 UTC m=+1151.345917207" watchObservedRunningTime="2026-09-08 20:44:58.096547093 +0000 UTC m=+1151.348028451"
Sep 08 20:45:00.140945 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:45:00.140913    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-78dd9f-29815005-df88w"]
Sep 08 20:45:00.144209 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:45:00.144193    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-78dd9f-29815005-df88w"
Sep 08 20:45:00.146573 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:45:00.146548    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2b2qs\""
Sep 08 20:45:00.153313 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:45:00.153286    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-78dd9f-29815005-df88w"]
Sep 08 20:45:00.155506 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:45:00.155332    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-78dd9f-29815005-df88w"
Sep 08 20:45:00.435088 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:45:00.435012    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-78dd9f-29815005-df88w"]
Sep 08 20:45:00.437701 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:45:00.437672    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaedcc132_20b4_4685_a5eb_e6c54772c8a6.slice/crio-2b6d7d6cc4112afd023b05de4be1ef880662d00bf7d23e76bfc4dc03a1e1b569 WatchSource:0}: Error finding container 2b6d7d6cc4112afd023b05de4be1ef880662d00bf7d23e76bfc4dc03a1e1b569: Status 404 returned error can't find the container with id 2b6d7d6cc4112afd023b05de4be1ef880662d00bf7d23e76bfc4dc03a1e1b569
Sep 08 20:45:01.092632 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:45:01.092602    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-78dd9f-29815005-df88w" event={"ID":"aedcc132-20b4-4685-a5eb-e6c54772c8a6","Type":"ContainerStarted","Data":"4f44221c280bf392b8ac789cef2790646eb86f5f1f1f565b5097c27cc32a1a65"}
Sep 08 20:45:01.092800 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:45:01.092654    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-78dd9f-29815005-df88w" event={"ID":"aedcc132-20b4-4685-a5eb-e6c54772c8a6","Type":"ContainerStarted","Data":"2b6d7d6cc4112afd023b05de4be1ef880662d00bf7d23e76bfc4dc03a1e1b569"}
Sep 08 20:45:01.106452 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:45:01.106399    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-78dd9f-29815005-df88w" podStartSLOduration=1.10637934 podStartE2EDuration="1.10637934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:45:01.106044516 +0000 UTC m=+1154.357525872" watchObservedRunningTime="2026-09-08 20:45:01.10637934 +0000 UTC m=+1154.357860796"
Sep 08 20:45:02.098642 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:45:02.098606    2552 generic.go:356] "Generic (PLEG): container finished" podID="aedcc132-20b4-4685-a5eb-e6c54772c8a6" containerID="4f44221c280bf392b8ac789cef2790646eb86f5f1f1f565b5097c27cc32a1a65" exitCode=0
Sep 08 20:45:02.099038 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:45:02.098695    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-78dd9f-29815005-df88w" event={"ID":"aedcc132-20b4-4685-a5eb-e6c54772c8a6","Type":"ContainerDied","Data":"4f44221c280bf392b8ac789cef2790646eb86f5f1f1f565b5097c27cc32a1a65"}
Sep 08 20:45:03.238952 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:45:03.238918    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-78dd9f-29815005-df88w"
Sep 08 20:45:04.087827 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:45:04.087792    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"
Sep 08 20:45:04.109118 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:45:04.109091    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-78dd9f-29815005-df88w"
Sep 08 20:45:04.109505 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:45:04.109124    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-78dd9f-29815005-df88w" event={"ID":"aedcc132-20b4-4685-a5eb-e6c54772c8a6","Type":"ContainerDied","Data":"2b6d7d6cc4112afd023b05de4be1ef880662d00bf7d23e76bfc4dc03a1e1b569"}
Sep 08 20:45:04.109505 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:45:04.109311    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b6d7d6cc4112afd023b05de4be1ef880662d00bf7d23e76bfc4dc03a1e1b569"
Sep 08 20:45:08.081289 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:45:08.081232    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59"
Sep 08 20:45:12.889976 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:45:12.889891    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:45:38.793253 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:45:38.793201    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:45:47.262390 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:45:47.262359    2552 scope.go:117] "RemoveContainer" containerID="a7505290047216be9e573d4e65bcdf58100a1bbaed6ac20e6395871c03752e2e"
Sep 08 20:45:47.270596 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:45:47.270576    2552 scope.go:117] "RemoveContainer" containerID="62aa163d8885ad976c62aab88b64ad0e3dcafa8b9d0f6fdc25acc6b563407e19"
Sep 08 20:45:51.997497 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:45:51.997460    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:46:10.287143 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:10.287105    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:46:23.987014 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:23.986977    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:46:29.313944 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.313907    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"]
Sep 08 20:46:29.314409 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.314202    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2" podUID="357d86f6-d540-4d8a-824e-be2a4a7a6716" containerName="maas-api" containerID="cri-o://0aa27413dedbab51a6685f21f84958ee465927e05a182e7329d12cb549327885" gracePeriod=30
Sep 08 20:46:29.372496 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.372461    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59"]
Sep 08 20:46:29.372761 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.372726    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59" podUID="8b11f91c-1280-46e6-ba91-faedc2bb61da" containerName="payload-pre-processing" containerID="cri-o://9be2ef1c4d3feb27750202c5f6b9279d7be851d0de403a3ad75bc82da36f334d" gracePeriod=30
Sep 08 20:46:29.490203 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.490165    2552 generic.go:356] "Generic (PLEG): container finished" podID="357d86f6-d540-4d8a-824e-be2a4a7a6716" containerID="0aa27413dedbab51a6685f21f84958ee465927e05a182e7329d12cb549327885" exitCode=0
Sep 08 20:46:29.490376 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.490206    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2" event={"ID":"357d86f6-d540-4d8a-824e-be2a4a7a6716","Type":"ContainerDied","Data":"0aa27413dedbab51a6685f21f84958ee465927e05a182e7329d12cb549327885"}
Sep 08 20:46:29.492136 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.492109    2552 generic.go:356] "Generic (PLEG): container finished" podID="8b11f91c-1280-46e6-ba91-faedc2bb61da" containerID="9be2ef1c4d3feb27750202c5f6b9279d7be851d0de403a3ad75bc82da36f334d" exitCode=0
Sep 08 20:46:29.492287 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.492145    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59" event={"ID":"8b11f91c-1280-46e6-ba91-faedc2bb61da","Type":"ContainerDied","Data":"9be2ef1c4d3feb27750202c5f6b9279d7be851d0de403a3ad75bc82da36f334d"}
Sep 08 20:46:29.617510 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.617484    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"
Sep 08 20:46:29.650688 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.650657    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59"
Sep 08 20:46:29.691292 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.691214    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/357d86f6-d540-4d8a-824e-be2a4a7a6716-maas-api-tls\") pod \"357d86f6-d540-4d8a-824e-be2a4a7a6716\" (UID: \"357d86f6-d540-4d8a-824e-be2a4a7a6716\") "
Sep 08 20:46:29.691422 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.691301    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qmmr\" (UniqueName: \"kubernetes.io/projected/357d86f6-d540-4d8a-824e-be2a4a7a6716-kube-api-access-2qmmr\") pod \"357d86f6-d540-4d8a-824e-be2a4a7a6716\" (UID: \"357d86f6-d540-4d8a-824e-be2a4a7a6716\") "
Sep 08 20:46:29.691422 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.691323    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b11f91c-1280-46e6-ba91-faedc2bb61da-plugins-config-volume\") pod \"8b11f91c-1280-46e6-ba91-faedc2bb61da\" (UID: \"8b11f91c-1280-46e6-ba91-faedc2bb61da\") "
Sep 08 20:46:29.691422 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.691373    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/357d86f6-d540-4d8a-824e-be2a4a7a6716-openshift-service-ca\") pod \"357d86f6-d540-4d8a-824e-be2a4a7a6716\" (UID: \"357d86f6-d540-4d8a-824e-be2a4a7a6716\") "
Sep 08 20:46:29.691422 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.691398    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/357d86f6-d540-4d8a-824e-be2a4a7a6716-metrics-certs\") pod \"357d86f6-d540-4d8a-824e-be2a4a7a6716\" (UID: \"357d86f6-d540-4d8a-824e-be2a4a7a6716\") "
Sep 08 20:46:29.691682 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.691469    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nft6n\" (UniqueName: \"kubernetes.io/projected/8b11f91c-1280-46e6-ba91-faedc2bb61da-kube-api-access-nft6n\") pod \"8b11f91c-1280-46e6-ba91-faedc2bb61da\" (UID: \"8b11f91c-1280-46e6-ba91-faedc2bb61da\") "
Sep 08 20:46:29.691772 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.691737    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b11f91c-1280-46e6-ba91-faedc2bb61da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b11f91c-1280-46e6-ba91-faedc2bb61da" (UID: "8b11f91c-1280-46e6-ba91-faedc2bb61da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:46:29.691861 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.691774    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357d86f6-d540-4d8a-824e-be2a4a7a6716-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "357d86f6-d540-4d8a-824e-be2a4a7a6716" (UID: "357d86f6-d540-4d8a-824e-be2a4a7a6716"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:46:29.693463 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.693425    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/357d86f6-d540-4d8a-824e-be2a4a7a6716-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "357d86f6-d540-4d8a-824e-be2a4a7a6716" (UID: "357d86f6-d540-4d8a-824e-be2a4a7a6716"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 20:46:29.693582 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.693473    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/357d86f6-d540-4d8a-824e-be2a4a7a6716-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "357d86f6-d540-4d8a-824e-be2a4a7a6716" (UID: "357d86f6-d540-4d8a-824e-be2a4a7a6716"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 20:46:29.693582 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.693487    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357d86f6-d540-4d8a-824e-be2a4a7a6716-kube-api-access-2qmmr" (OuterVolumeSpecName: "kube-api-access-2qmmr") pod "357d86f6-d540-4d8a-824e-be2a4a7a6716" (UID: "357d86f6-d540-4d8a-824e-be2a4a7a6716"). InnerVolumeSpecName "kube-api-access-2qmmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:46:29.693582 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.693528    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b11f91c-1280-46e6-ba91-faedc2bb61da-kube-api-access-nft6n" (OuterVolumeSpecName: "kube-api-access-nft6n") pod "8b11f91c-1280-46e6-ba91-faedc2bb61da" (UID: "8b11f91c-1280-46e6-ba91-faedc2bb61da"). InnerVolumeSpecName "kube-api-access-nft6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:46:29.792415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.792375    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/357d86f6-d540-4d8a-824e-be2a4a7a6716-openshift-service-ca\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:46:29.792415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.792406    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/357d86f6-d540-4d8a-824e-be2a4a7a6716-metrics-certs\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:46:29.792415 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.792420    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nft6n\" (UniqueName: \"kubernetes.io/projected/8b11f91c-1280-46e6-ba91-faedc2bb61da-kube-api-access-nft6n\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:46:29.792660 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.792431    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/357d86f6-d540-4d8a-824e-be2a4a7a6716-maas-api-tls\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:46:29.792660 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.792443    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qmmr\" (UniqueName: \"kubernetes.io/projected/357d86f6-d540-4d8a-824e-be2a4a7a6716-kube-api-access-2qmmr\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:46:29.792660 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:29.792457    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b11f91c-1280-46e6-ba91-faedc2bb61da-plugins-config-volume\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:46:30.497940 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:30.497912    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59"
Sep 08 20:46:30.498413 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:30.497949    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59" event={"ID":"8b11f91c-1280-46e6-ba91-faedc2bb61da","Type":"ContainerDied","Data":"c6f7782f19cea90ef76902a4d9be78403ec8703f4b4a21fc6c49b394b2096368"}
Sep 08 20:46:30.498413 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:30.498013    2552 scope.go:117] "RemoveContainer" containerID="9be2ef1c4d3feb27750202c5f6b9279d7be851d0de403a3ad75bc82da36f334d"
Sep 08 20:46:30.499461 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:30.499435    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"
Sep 08 20:46:30.499520 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:30.499458    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2" event={"ID":"357d86f6-d540-4d8a-824e-be2a4a7a6716","Type":"ContainerDied","Data":"10d8d8d0c886e91321f3347922e0d9678e68f9409321e3feef9befe9cb8be536"}
Sep 08 20:46:30.507894 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:30.507875    2552 scope.go:117] "RemoveContainer" containerID="0aa27413dedbab51a6685f21f84958ee465927e05a182e7329d12cb549327885"
Sep 08 20:46:30.520654 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:30.520630    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59"]
Sep 08 20:46:30.524325 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:30.524302    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e0dc0027-7d9bb5cccd-7fb59"]
Sep 08 20:46:30.534412 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:30.534391    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"]
Sep 08 20:46:30.538439 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:30.538420    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0dc0027-58c6764d55-pplg2"]
Sep 08 20:46:31.224643 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:31.224599    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="357d86f6-d540-4d8a-824e-be2a4a7a6716" path="/var/lib/kubelet/pods/357d86f6-d540-4d8a-824e-be2a4a7a6716/volumes"
Sep 08 20:46:31.225051 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:31.225031    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b11f91c-1280-46e6-ba91-faedc2bb61da" path="/var/lib/kubelet/pods/8b11f91c-1280-46e6-ba91-faedc2bb61da/volumes"
Sep 08 20:46:44.196020 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:46:44.195925    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:47:00.989456 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:47:00.989412    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:47:18.303585 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:47:18.303546    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:47:33.787635 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:47:33.787602    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:47:50.291694 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:47:50.291659    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:48:07.793957 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:07.793924    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:48:21.792305 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:21.792266    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:48:34.763317 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.763281    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"]
Sep 08 20:48:34.763778 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.763763    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b11f91c-1280-46e6-ba91-faedc2bb61da" containerName="payload-pre-processing"
Sep 08 20:48:34.763825 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.763780    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b11f91c-1280-46e6-ba91-faedc2bb61da" containerName="payload-pre-processing"
Sep 08 20:48:34.763825 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.763801    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357d86f6-d540-4d8a-824e-be2a4a7a6716" containerName="maas-api"
Sep 08 20:48:34.763825 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.763810    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="357d86f6-d540-4d8a-824e-be2a4a7a6716" containerName="maas-api"
Sep 08 20:48:34.763825 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.763822    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aedcc132-20b4-4685-a5eb-e6c54772c8a6" containerName="cleanup"
Sep 08 20:48:34.763950 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.763832    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="aedcc132-20b4-4685-a5eb-e6c54772c8a6" containerName="cleanup"
Sep 08 20:48:34.763950 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.763916    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="357d86f6-d540-4d8a-824e-be2a4a7a6716" containerName="maas-api"
Sep 08 20:48:34.763950 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.763930    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b11f91c-1280-46e6-ba91-faedc2bb61da" containerName="payload-pre-processing"
Sep 08 20:48:34.763950 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.763940    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="aedcc132-20b4-4685-a5eb-e6c54772c8a6" containerName="cleanup"
Sep 08 20:48:34.767424 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.767402    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"
Sep 08 20:48:34.769869 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.769850    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ce2cc8-serving-cert\""
Sep 08 20:48:34.777524 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.777504    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"]
Sep 08 20:48:34.870141 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.870107    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbff241a-9952-44c5-81ea-5b9c0b864e05-metrics-certs\") pod \"maas-api-e2e-api-a-ce2cc8-755765485-v4bgs\" (UID: \"dbff241a-9952-44c5-81ea-5b9c0b864e05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"
Sep 08 20:48:34.870357 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.870212    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbff241a-9952-44c5-81ea-5b9c0b864e05-openshift-service-ca\") pod \"maas-api-e2e-api-a-ce2cc8-755765485-v4bgs\" (UID: \"dbff241a-9952-44c5-81ea-5b9c0b864e05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"
Sep 08 20:48:34.870357 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.870295    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d9jl\" (UniqueName: \"kubernetes.io/projected/dbff241a-9952-44c5-81ea-5b9c0b864e05-kube-api-access-5d9jl\") pod \"maas-api-e2e-api-a-ce2cc8-755765485-v4bgs\" (UID: \"dbff241a-9952-44c5-81ea-5b9c0b864e05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"
Sep 08 20:48:34.870357 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.870324    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbff241a-9952-44c5-81ea-5b9c0b864e05-maas-api-tls\") pod \"maas-api-e2e-api-a-ce2cc8-755765485-v4bgs\" (UID: \"dbff241a-9952-44c5-81ea-5b9c0b864e05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"
Sep 08 20:48:34.971647 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.971614    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d9jl\" (UniqueName: \"kubernetes.io/projected/dbff241a-9952-44c5-81ea-5b9c0b864e05-kube-api-access-5d9jl\") pod \"maas-api-e2e-api-a-ce2cc8-755765485-v4bgs\" (UID: \"dbff241a-9952-44c5-81ea-5b9c0b864e05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"
Sep 08 20:48:34.971904 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.971667    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbff241a-9952-44c5-81ea-5b9c0b864e05-maas-api-tls\") pod \"maas-api-e2e-api-a-ce2cc8-755765485-v4bgs\" (UID: \"dbff241a-9952-44c5-81ea-5b9c0b864e05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"
Sep 08 20:48:34.971904 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.971700    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbff241a-9952-44c5-81ea-5b9c0b864e05-metrics-certs\") pod \"maas-api-e2e-api-a-ce2cc8-755765485-v4bgs\" (UID: \"dbff241a-9952-44c5-81ea-5b9c0b864e05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"
Sep 08 20:48:34.971904 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.971779    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbff241a-9952-44c5-81ea-5b9c0b864e05-openshift-service-ca\") pod \"maas-api-e2e-api-a-ce2cc8-755765485-v4bgs\" (UID: \"dbff241a-9952-44c5-81ea-5b9c0b864e05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"
Sep 08 20:48:34.972533 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.972499    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbff241a-9952-44c5-81ea-5b9c0b864e05-openshift-service-ca\") pod \"maas-api-e2e-api-a-ce2cc8-755765485-v4bgs\" (UID: \"dbff241a-9952-44c5-81ea-5b9c0b864e05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"
Sep 08 20:48:34.974191 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.974166    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbff241a-9952-44c5-81ea-5b9c0b864e05-metrics-certs\") pod \"maas-api-e2e-api-a-ce2cc8-755765485-v4bgs\" (UID: \"dbff241a-9952-44c5-81ea-5b9c0b864e05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"
Sep 08 20:48:34.974298 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.974206    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbff241a-9952-44c5-81ea-5b9c0b864e05-maas-api-tls\") pod \"maas-api-e2e-api-a-ce2cc8-755765485-v4bgs\" (UID: \"dbff241a-9952-44c5-81ea-5b9c0b864e05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"
Sep 08 20:48:34.981300 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:34.981276    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d9jl\" (UniqueName: \"kubernetes.io/projected/dbff241a-9952-44c5-81ea-5b9c0b864e05-kube-api-access-5d9jl\") pod \"maas-api-e2e-api-a-ce2cc8-755765485-v4bgs\" (UID: \"dbff241a-9952-44c5-81ea-5b9c0b864e05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"
Sep 08 20:48:35.080578 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:35.080499    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"
Sep 08 20:48:35.359605 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:35.359577    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"]
Sep 08 20:48:35.361953 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:48:35.361927    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbff241a_9952_44c5_81ea_5b9c0b864e05.slice/crio-da85b68bc13c7f07dfc42217c1e92d91ae58dbfb43a44468cb3a9bd13e46084f WatchSource:0}: Error finding container da85b68bc13c7f07dfc42217c1e92d91ae58dbfb43a44468cb3a9bd13e46084f: Status 404 returned error can't find the container with id da85b68bc13c7f07dfc42217c1e92d91ae58dbfb43a44468cb3a9bd13e46084f
Sep 08 20:48:35.363483 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:35.363460    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 20:48:36.055979 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:36.055940    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs" event={"ID":"dbff241a-9952-44c5-81ea-5b9c0b864e05","Type":"ContainerStarted","Data":"9f0190094a862866b0898fdb0a61041f3f651880c5bead10ac2fd7c6315e4b2d"}
Sep 08 20:48:36.056381 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:36.056004    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs" event={"ID":"dbff241a-9952-44c5-81ea-5b9c0b864e05","Type":"ContainerStarted","Data":"da85b68bc13c7f07dfc42217c1e92d91ae58dbfb43a44468cb3a9bd13e46084f"}
Sep 08 20:48:36.056381 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:36.056115    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"
Sep 08 20:48:36.074001 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:36.073956    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs" podStartSLOduration=1.5126753389999998 podStartE2EDuration="2.073942451s" totalImagesPullingTime="561.267112ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:48:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:48:36.072013059 +0000 UTC m=+1369.323494417" watchObservedRunningTime="2026-09-08 20:48:36.073942451 +0000 UTC m=+1369.325423809"
Sep 08 20:48:37.090902 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:37.090865    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:48:42.066712 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:42.066683    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"
Sep 08 20:48:54.604996 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:48:54.604964    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:49:03.881793 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:03.881755    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"]
Sep 08 20:49:03.885739 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:03.885715    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"
Sep 08 20:49:03.888074 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:03.888051    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-792dc6-serving-cert\""
Sep 08 20:49:03.894313 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:03.894225    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"]
Sep 08 20:49:04.035725 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.035682    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b50a1e4-735a-4467-814c-af13db02c3cc-openshift-service-ca\") pod \"maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc\" (UID: \"4b50a1e4-735a-4467-814c-af13db02c3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"
Sep 08 20:49:04.035725 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.035731    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slllc\" (UniqueName: \"kubernetes.io/projected/4b50a1e4-735a-4467-814c-af13db02c3cc-kube-api-access-slllc\") pod \"maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc\" (UID: \"4b50a1e4-735a-4467-814c-af13db02c3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"
Sep 08 20:49:04.035941 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.035804    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b50a1e4-735a-4467-814c-af13db02c3cc-metrics-certs\") pod \"maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc\" (UID: \"4b50a1e4-735a-4467-814c-af13db02c3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"
Sep 08 20:49:04.035980 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.035942    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b50a1e4-735a-4467-814c-af13db02c3cc-maas-api-tls\") pod \"maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc\" (UID: \"4b50a1e4-735a-4467-814c-af13db02c3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"
Sep 08 20:49:04.137105 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.137023    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b50a1e4-735a-4467-814c-af13db02c3cc-metrics-certs\") pod \"maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc\" (UID: \"4b50a1e4-735a-4467-814c-af13db02c3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"
Sep 08 20:49:04.137276 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.137114    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b50a1e4-735a-4467-814c-af13db02c3cc-maas-api-tls\") pod \"maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc\" (UID: \"4b50a1e4-735a-4467-814c-af13db02c3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"
Sep 08 20:49:04.137276 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.137137    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b50a1e4-735a-4467-814c-af13db02c3cc-openshift-service-ca\") pod \"maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc\" (UID: \"4b50a1e4-735a-4467-814c-af13db02c3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"
Sep 08 20:49:04.137276 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.137153    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slllc\" (UniqueName: \"kubernetes.io/projected/4b50a1e4-735a-4467-814c-af13db02c3cc-kube-api-access-slllc\") pod \"maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc\" (UID: \"4b50a1e4-735a-4467-814c-af13db02c3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"
Sep 08 20:49:04.137852 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.137831    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b50a1e4-735a-4467-814c-af13db02c3cc-openshift-service-ca\") pod \"maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc\" (UID: \"4b50a1e4-735a-4467-814c-af13db02c3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"
Sep 08 20:49:04.139480 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.139456    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b50a1e4-735a-4467-814c-af13db02c3cc-metrics-certs\") pod \"maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc\" (UID: \"4b50a1e4-735a-4467-814c-af13db02c3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"
Sep 08 20:49:04.139574 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.139497    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b50a1e4-735a-4467-814c-af13db02c3cc-maas-api-tls\") pod \"maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc\" (UID: \"4b50a1e4-735a-4467-814c-af13db02c3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"
Sep 08 20:49:04.144930 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.144909    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slllc\" (UniqueName: \"kubernetes.io/projected/4b50a1e4-735a-4467-814c-af13db02c3cc-kube-api-access-slllc\") pod \"maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc\" (UID: \"4b50a1e4-735a-4467-814c-af13db02c3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"
Sep 08 20:49:04.198876 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.198844    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"
Sep 08 20:49:04.468533 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.468509    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"]
Sep 08 20:49:04.470697 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:49:04.470668    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b50a1e4_735a_4467_814c_af13db02c3cc.slice/crio-cc1365fc91145fe615b4986044b1151bfcc20d6626b68cd5d2080e0ca528b37f WatchSource:0}: Error finding container cc1365fc91145fe615b4986044b1151bfcc20d6626b68cd5d2080e0ca528b37f: Status 404 returned error can't find the container with id cc1365fc91145fe615b4986044b1151bfcc20d6626b68cd5d2080e0ca528b37f
Sep 08 20:49:04.506616 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.506589    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg"]
Sep 08 20:49:04.511495 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.511465    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg"
Sep 08 20:49:04.513924 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.513900    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-792dc6-dockercfg-sdz96\""
Sep 08 20:49:04.514040 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.513947    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-792dc6\""
Sep 08 20:49:04.523396 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.523366    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg"]
Sep 08 20:49:04.533436 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.533411    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl"]
Sep 08 20:49:04.537083 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.537064    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl"
Sep 08 20:49:04.546703 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.546679    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl"]
Sep 08 20:49:04.641484 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.641453    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9lvc\" (UniqueName: \"kubernetes.io/projected/15523e6f-2ba7-430f-8376-25e1c4db7ae2-kube-api-access-k9lvc\") pod \"payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg\" (UID: \"15523e6f-2ba7-430f-8376-25e1c4db7ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg"
Sep 08 20:49:04.641634 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.641520    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72hhf\" (UniqueName: \"kubernetes.io/projected/ccb9ef04-59dc-410d-a463-d6dd13b925dd-kube-api-access-72hhf\") pod \"payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl\" (UID: \"ccb9ef04-59dc-410d-a463-d6dd13b925dd\") " pod="openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl"
Sep 08 20:49:04.641634 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.641581    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15523e6f-2ba7-430f-8376-25e1c4db7ae2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg\" (UID: \"15523e6f-2ba7-430f-8376-25e1c4db7ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg"
Sep 08 20:49:04.641721 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.641702    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb9ef04-59dc-410d-a463-d6dd13b925dd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl\" (UID: \"ccb9ef04-59dc-410d-a463-d6dd13b925dd\") " pod="openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl"
Sep 08 20:49:04.742534 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.742447    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72hhf\" (UniqueName: \"kubernetes.io/projected/ccb9ef04-59dc-410d-a463-d6dd13b925dd-kube-api-access-72hhf\") pod \"payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl\" (UID: \"ccb9ef04-59dc-410d-a463-d6dd13b925dd\") " pod="openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl"
Sep 08 20:49:04.742534 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.742500    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15523e6f-2ba7-430f-8376-25e1c4db7ae2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg\" (UID: \"15523e6f-2ba7-430f-8376-25e1c4db7ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg"
Sep 08 20:49:04.742732 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.742550    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb9ef04-59dc-410d-a463-d6dd13b925dd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl\" (UID: \"ccb9ef04-59dc-410d-a463-d6dd13b925dd\") " pod="openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl"
Sep 08 20:49:04.742732 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.742574    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9lvc\" (UniqueName: \"kubernetes.io/projected/15523e6f-2ba7-430f-8376-25e1c4db7ae2-kube-api-access-k9lvc\") pod \"payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg\" (UID: \"15523e6f-2ba7-430f-8376-25e1c4db7ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg"
Sep 08 20:49:04.743490 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.743469    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb9ef04-59dc-410d-a463-d6dd13b925dd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl\" (UID: \"ccb9ef04-59dc-410d-a463-d6dd13b925dd\") " pod="openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl"
Sep 08 20:49:04.743537 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.743469    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15523e6f-2ba7-430f-8376-25e1c4db7ae2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg\" (UID: \"15523e6f-2ba7-430f-8376-25e1c4db7ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg"
Sep 08 20:49:04.750613 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.750588    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9lvc\" (UniqueName: \"kubernetes.io/projected/15523e6f-2ba7-430f-8376-25e1c4db7ae2-kube-api-access-k9lvc\") pod \"payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg\" (UID: \"15523e6f-2ba7-430f-8376-25e1c4db7ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg"
Sep 08 20:49:04.750729 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.750666    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72hhf\" (UniqueName: \"kubernetes.io/projected/ccb9ef04-59dc-410d-a463-d6dd13b925dd-kube-api-access-72hhf\") pod \"payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl\" (UID: \"ccb9ef04-59dc-410d-a463-d6dd13b925dd\") " pod="openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl"
Sep 08 20:49:04.828895 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.828876    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg"
Sep 08 20:49:04.847559 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:04.847534    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl"
Sep 08 20:49:05.102930 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:05.102898    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg"]
Sep 08 20:49:05.105711 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:49:05.105677    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15523e6f_2ba7_430f_8376_25e1c4db7ae2.slice/crio-caf4595f068290b45fc5fedc27dab574f7243d6b7c8384caa422c9fa128447ee WatchSource:0}: Error finding container caf4595f068290b45fc5fedc27dab574f7243d6b7c8384caa422c9fa128447ee: Status 404 returned error can't find the container with id caf4595f068290b45fc5fedc27dab574f7243d6b7c8384caa422c9fa128447ee
Sep 08 20:49:05.137463 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:05.137440    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl"]
Sep 08 20:49:05.141377 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:49:05.141353    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb9ef04_59dc_410d_a463_d6dd13b925dd.slice/crio-7ff0e6345c5201d10168763d93dfa76cddcb722904d4e1e6ef9bad7e13aea988 WatchSource:0}: Error finding container 7ff0e6345c5201d10168763d93dfa76cddcb722904d4e1e6ef9bad7e13aea988: Status 404 returned error can't find the container with id 7ff0e6345c5201d10168763d93dfa76cddcb722904d4e1e6ef9bad7e13aea988
Sep 08 20:49:05.187461 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:05.187435    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl" event={"ID":"ccb9ef04-59dc-410d-a463-d6dd13b925dd","Type":"ContainerStarted","Data":"7ff0e6345c5201d10168763d93dfa76cddcb722904d4e1e6ef9bad7e13aea988"}
Sep 08 20:49:05.189203 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:05.189161    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc" event={"ID":"4b50a1e4-735a-4467-814c-af13db02c3cc","Type":"ContainerStarted","Data":"ee97d6d5d33cd171d0f822a9921ae32b805a54118358df697b1b02e543526a74"}
Sep 08 20:49:05.189332 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:05.189215    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"
Sep 08 20:49:05.189332 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:05.189255    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc" event={"ID":"4b50a1e4-735a-4467-814c-af13db02c3cc","Type":"ContainerStarted","Data":"cc1365fc91145fe615b4986044b1151bfcc20d6626b68cd5d2080e0ca528b37f"}
Sep 08 20:49:05.190603 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:05.190578    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg" event={"ID":"15523e6f-2ba7-430f-8376-25e1c4db7ae2","Type":"ContainerStarted","Data":"caf4595f068290b45fc5fedc27dab574f7243d6b7c8384caa422c9fa128447ee"}
Sep 08 20:49:05.205776 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:05.205735    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc" podStartSLOduration=1.870271764 podStartE2EDuration="2.205721196s" totalImagesPullingTime="335.449432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:49:05.204868144 +0000 UTC m=+1398.456349503" watchObservedRunningTime="2026-09-08 20:49:05.205721196 +0000 UTC m=+1398.457202553"
Sep 08 20:49:06.196901 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:06.196857    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg" event={"ID":"15523e6f-2ba7-430f-8376-25e1c4db7ae2","Type":"ContainerStarted","Data":"b6b5ed7b7dd500d802dff063cfbdb61108c3dd6d3c4f11bb451136553d27cadb"}
Sep 08 20:49:06.197397 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:06.197037    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg"
Sep 08 20:49:06.198656 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:06.198631    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl" event={"ID":"ccb9ef04-59dc-410d-a463-d6dd13b925dd","Type":"ContainerStarted","Data":"d765af8be11c38554ca23b12861ecc05e2c53a071b63fddf583651782caab58f"}
Sep 08 20:49:06.198783 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:06.198675    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl"
Sep 08 20:49:06.212857 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:06.212815    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg" podStartSLOduration=2.212800913 podStartE2EDuration="2.212800913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:49:06.212401033 +0000 UTC m=+1399.463882390" watchObservedRunningTime="2026-09-08 20:49:06.212800913 +0000 UTC m=+1399.464282270"
Sep 08 20:49:06.228131 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:06.228085    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl" podStartSLOduration=2.228073169 podStartE2EDuration="2.228073169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:49:06.227230155 +0000 UTC m=+1399.478711515" watchObservedRunningTime="2026-09-08 20:49:06.228073169 +0000 UTC m=+1399.479554577"
Sep 08 20:49:11.204631 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:11.204600    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"
Sep 08 20:49:11.899875 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:11.899840    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:49:17.206170 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:17.206137    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl"
Sep 08 20:49:17.206652 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:17.206211    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg"
Sep 08 20:49:31.496149 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:31.496110    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:49:43.196614 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:43.196531    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:49:46.680083 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:46.680051    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"]
Sep 08 20:49:46.680595 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:46.680292    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs" podUID="dbff241a-9952-44c5-81ea-5b9c0b864e05" containerName="maas-api" containerID="cri-o://9f0190094a862866b0898fdb0a61041f3f651880c5bead10ac2fd7c6315e4b2d" gracePeriod=30
Sep 08 20:49:46.924547 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:46.924519    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"
Sep 08 20:49:47.008682 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:47.008598    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbff241a-9952-44c5-81ea-5b9c0b864e05-maas-api-tls\") pod \"dbff241a-9952-44c5-81ea-5b9c0b864e05\" (UID: \"dbff241a-9952-44c5-81ea-5b9c0b864e05\") "
Sep 08 20:49:47.008682 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:47.008645    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbff241a-9952-44c5-81ea-5b9c0b864e05-metrics-certs\") pod \"dbff241a-9952-44c5-81ea-5b9c0b864e05\" (UID: \"dbff241a-9952-44c5-81ea-5b9c0b864e05\") "
Sep 08 20:49:47.008909 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:47.008718    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5d9jl\" (UniqueName: \"kubernetes.io/projected/dbff241a-9952-44c5-81ea-5b9c0b864e05-kube-api-access-5d9jl\") pod \"dbff241a-9952-44c5-81ea-5b9c0b864e05\" (UID: \"dbff241a-9952-44c5-81ea-5b9c0b864e05\") "
Sep 08 20:49:47.008909 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:47.008797    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbff241a-9952-44c5-81ea-5b9c0b864e05-openshift-service-ca\") pod \"dbff241a-9952-44c5-81ea-5b9c0b864e05\" (UID: \"dbff241a-9952-44c5-81ea-5b9c0b864e05\") "
Sep 08 20:49:47.009229 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:47.009192    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbff241a-9952-44c5-81ea-5b9c0b864e05-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dbff241a-9952-44c5-81ea-5b9c0b864e05" (UID: "dbff241a-9952-44c5-81ea-5b9c0b864e05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:49:47.010766 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:47.010740    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbff241a-9952-44c5-81ea-5b9c0b864e05-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dbff241a-9952-44c5-81ea-5b9c0b864e05" (UID: "dbff241a-9952-44c5-81ea-5b9c0b864e05"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 20:49:47.010865 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:47.010794    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbff241a-9952-44c5-81ea-5b9c0b864e05-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dbff241a-9952-44c5-81ea-5b9c0b864e05" (UID: "dbff241a-9952-44c5-81ea-5b9c0b864e05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 20:49:47.010865 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:47.010819    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbff241a-9952-44c5-81ea-5b9c0b864e05-kube-api-access-5d9jl" (OuterVolumeSpecName: "kube-api-access-5d9jl") pod "dbff241a-9952-44c5-81ea-5b9c0b864e05" (UID: "dbff241a-9952-44c5-81ea-5b9c0b864e05"). InnerVolumeSpecName "kube-api-access-5d9jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:49:47.109610 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:47.109583    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d9jl\" (UniqueName: \"kubernetes.io/projected/dbff241a-9952-44c5-81ea-5b9c0b864e05-kube-api-access-5d9jl\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:49:47.109610 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:47.109606    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbff241a-9952-44c5-81ea-5b9c0b864e05-openshift-service-ca\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:49:47.109610 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:47.109616    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbff241a-9952-44c5-81ea-5b9c0b864e05-maas-api-tls\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:49:47.109844 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:47.109625    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbff241a-9952-44c5-81ea-5b9c0b864e05-metrics-certs\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:49:47.405259 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:47.405205    2552 generic.go:356] "Generic (PLEG): container finished" podID="dbff241a-9952-44c5-81ea-5b9c0b864e05" containerID="9f0190094a862866b0898fdb0a61041f3f651880c5bead10ac2fd7c6315e4b2d" exitCode=0
Sep 08 20:49:47.405452 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:47.405277    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"
Sep 08 20:49:47.405452 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:47.405274    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs" event={"ID":"dbff241a-9952-44c5-81ea-5b9c0b864e05","Type":"ContainerDied","Data":"9f0190094a862866b0898fdb0a61041f3f651880c5bead10ac2fd7c6315e4b2d"}
Sep 08 20:49:47.405452 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:47.405372    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs" event={"ID":"dbff241a-9952-44c5-81ea-5b9c0b864e05","Type":"ContainerDied","Data":"da85b68bc13c7f07dfc42217c1e92d91ae58dbfb43a44468cb3a9bd13e46084f"}
Sep 08 20:49:47.405452 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:47.405404    2552 scope.go:117] "RemoveContainer" containerID="9f0190094a862866b0898fdb0a61041f3f651880c5bead10ac2fd7c6315e4b2d"
Sep 08 20:49:47.414390 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:47.414361    2552 scope.go:117] "RemoveContainer" containerID="9f0190094a862866b0898fdb0a61041f3f651880c5bead10ac2fd7c6315e4b2d"
Sep 08 20:49:47.414684 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:49:47.414661    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0190094a862866b0898fdb0a61041f3f651880c5bead10ac2fd7c6315e4b2d\": container with ID starting with 9f0190094a862866b0898fdb0a61041f3f651880c5bead10ac2fd7c6315e4b2d not found: ID does not exist" containerID="9f0190094a862866b0898fdb0a61041f3f651880c5bead10ac2fd7c6315e4b2d"
Sep 08 20:49:47.414789 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:47.414689    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0190094a862866b0898fdb0a61041f3f651880c5bead10ac2fd7c6315e4b2d"} err="failed to get container status \"9f0190094a862866b0898fdb0a61041f3f651880c5bead10ac2fd7c6315e4b2d\": rpc error: code = NotFound desc = could not find container \"9f0190094a862866b0898fdb0a61041f3f651880c5bead10ac2fd7c6315e4b2d\": container with ID starting with 9f0190094a862866b0898fdb0a61041f3f651880c5bead10ac2fd7c6315e4b2d not found: ID does not exist"
Sep 08 20:49:47.425870 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:47.425841    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"]
Sep 08 20:49:47.429372 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:47.429350    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ce2cc8-755765485-v4bgs"]
Sep 08 20:49:49.224798 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:49.224763    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbff241a-9952-44c5-81ea-5b9c0b864e05" path="/var/lib/kubelet/pods/dbff241a-9952-44c5-81ea-5b9c0b864e05/volumes"
Sep 08 20:49:59.686996 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:49:59.686961    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:50:09.295159 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.295115    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"]
Sep 08 20:50:09.295650 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.295345    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc" podUID="4b50a1e4-735a-4467-814c-af13db02c3cc" containerName="maas-api" containerID="cri-o://ee97d6d5d33cd171d0f822a9921ae32b805a54118358df697b1b02e543526a74" gracePeriod=30
Sep 08 20:50:09.351482 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.351445    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl"]
Sep 08 20:50:09.351732 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.351703    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl" podUID="ccb9ef04-59dc-410d-a463-d6dd13b925dd" containerName="payload-processing" containerID="cri-o://d765af8be11c38554ca23b12861ecc05e2c53a071b63fddf583651782caab58f" gracePeriod=30
Sep 08 20:50:09.357725 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.357696    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg"]
Sep 08 20:50:09.357943 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.357898    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg" podUID="15523e6f-2ba7-430f-8376-25e1c4db7ae2" containerName="payload-pre-processing" containerID="cri-o://b6b5ed7b7dd500d802dff063cfbdb61108c3dd6d3c4f11bb451136553d27cadb" gracePeriod=30
Sep 08 20:50:09.506890 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.506849    2552 generic.go:356] "Generic (PLEG): container finished" podID="ccb9ef04-59dc-410d-a463-d6dd13b925dd" containerID="d765af8be11c38554ca23b12861ecc05e2c53a071b63fddf583651782caab58f" exitCode=0
Sep 08 20:50:09.507072 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.506981    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl" event={"ID":"ccb9ef04-59dc-410d-a463-d6dd13b925dd","Type":"ContainerDied","Data":"d765af8be11c38554ca23b12861ecc05e2c53a071b63fddf583651782caab58f"}
Sep 08 20:50:09.508986 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.508948    2552 generic.go:356] "Generic (PLEG): container finished" podID="4b50a1e4-735a-4467-814c-af13db02c3cc" containerID="ee97d6d5d33cd171d0f822a9921ae32b805a54118358df697b1b02e543526a74" exitCode=0
Sep 08 20:50:09.509116 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.509033    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc" event={"ID":"4b50a1e4-735a-4467-814c-af13db02c3cc","Type":"ContainerDied","Data":"ee97d6d5d33cd171d0f822a9921ae32b805a54118358df697b1b02e543526a74"}
Sep 08 20:50:09.510871 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.510834    2552 generic.go:356] "Generic (PLEG): container finished" podID="15523e6f-2ba7-430f-8376-25e1c4db7ae2" containerID="b6b5ed7b7dd500d802dff063cfbdb61108c3dd6d3c4f11bb451136553d27cadb" exitCode=0
Sep 08 20:50:09.510986 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.510865    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg" event={"ID":"15523e6f-2ba7-430f-8376-25e1c4db7ae2","Type":"ContainerDied","Data":"b6b5ed7b7dd500d802dff063cfbdb61108c3dd6d3c4f11bb451136553d27cadb"}
Sep 08 20:50:09.646525 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.646492    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl"
Sep 08 20:50:09.649478 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.649458    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"
Sep 08 20:50:09.653387 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.653369    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg"
Sep 08 20:50:09.808396 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.808314    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b50a1e4-735a-4467-814c-af13db02c3cc-metrics-certs\") pod \"4b50a1e4-735a-4467-814c-af13db02c3cc\" (UID: \"4b50a1e4-735a-4467-814c-af13db02c3cc\") "
Sep 08 20:50:09.808396 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.808347    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b50a1e4-735a-4467-814c-af13db02c3cc-maas-api-tls\") pod \"4b50a1e4-735a-4467-814c-af13db02c3cc\" (UID: \"4b50a1e4-735a-4467-814c-af13db02c3cc\") "
Sep 08 20:50:09.808583 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.808405    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb9ef04-59dc-410d-a463-d6dd13b925dd-plugins-config-volume\") pod \"ccb9ef04-59dc-410d-a463-d6dd13b925dd\" (UID: \"ccb9ef04-59dc-410d-a463-d6dd13b925dd\") "
Sep 08 20:50:09.808583 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.808562    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72hhf\" (UniqueName: \"kubernetes.io/projected/ccb9ef04-59dc-410d-a463-d6dd13b925dd-kube-api-access-72hhf\") pod \"ccb9ef04-59dc-410d-a463-d6dd13b925dd\" (UID: \"ccb9ef04-59dc-410d-a463-d6dd13b925dd\") "
Sep 08 20:50:09.808687 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.808594    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15523e6f-2ba7-430f-8376-25e1c4db7ae2-plugins-config-volume\") pod \"15523e6f-2ba7-430f-8376-25e1c4db7ae2\" (UID: \"15523e6f-2ba7-430f-8376-25e1c4db7ae2\") "
Sep 08 20:50:09.808687 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.808636    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slllc\" (UniqueName: \"kubernetes.io/projected/4b50a1e4-735a-4467-814c-af13db02c3cc-kube-api-access-slllc\") pod \"4b50a1e4-735a-4467-814c-af13db02c3cc\" (UID: \"4b50a1e4-735a-4467-814c-af13db02c3cc\") "
Sep 08 20:50:09.808798 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.808703    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b50a1e4-735a-4467-814c-af13db02c3cc-openshift-service-ca\") pod \"4b50a1e4-735a-4467-814c-af13db02c3cc\" (UID: \"4b50a1e4-735a-4467-814c-af13db02c3cc\") "
Sep 08 20:50:09.808798 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.808736    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9lvc\" (UniqueName: \"kubernetes.io/projected/15523e6f-2ba7-430f-8376-25e1c4db7ae2-kube-api-access-k9lvc\") pod \"15523e6f-2ba7-430f-8376-25e1c4db7ae2\" (UID: \"15523e6f-2ba7-430f-8376-25e1c4db7ae2\") "
Sep 08 20:50:09.808912 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.808783    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb9ef04-59dc-410d-a463-d6dd13b925dd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ccb9ef04-59dc-410d-a463-d6dd13b925dd" (UID: "ccb9ef04-59dc-410d-a463-d6dd13b925dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:50:09.809029 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.809008    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb9ef04-59dc-410d-a463-d6dd13b925dd-plugins-config-volume\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:50:09.809220 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.809188    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15523e6f-2ba7-430f-8376-25e1c4db7ae2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "15523e6f-2ba7-430f-8376-25e1c4db7ae2" (UID: "15523e6f-2ba7-430f-8376-25e1c4db7ae2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:50:09.809330 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.809214    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b50a1e4-735a-4467-814c-af13db02c3cc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4b50a1e4-735a-4467-814c-af13db02c3cc" (UID: "4b50a1e4-735a-4467-814c-af13db02c3cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:50:09.810739 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.810714    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b50a1e4-735a-4467-814c-af13db02c3cc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4b50a1e4-735a-4467-814c-af13db02c3cc" (UID: "4b50a1e4-735a-4467-814c-af13db02c3cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 20:50:09.810739 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.810728    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb9ef04-59dc-410d-a463-d6dd13b925dd-kube-api-access-72hhf" (OuterVolumeSpecName: "kube-api-access-72hhf") pod "ccb9ef04-59dc-410d-a463-d6dd13b925dd" (UID: "ccb9ef04-59dc-410d-a463-d6dd13b925dd"). InnerVolumeSpecName "kube-api-access-72hhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:50:09.810877 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.810750    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b50a1e4-735a-4467-814c-af13db02c3cc-kube-api-access-slllc" (OuterVolumeSpecName: "kube-api-access-slllc") pod "4b50a1e4-735a-4467-814c-af13db02c3cc" (UID: "4b50a1e4-735a-4467-814c-af13db02c3cc"). InnerVolumeSpecName "kube-api-access-slllc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:50:09.810967 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.810950    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b50a1e4-735a-4467-814c-af13db02c3cc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4b50a1e4-735a-4467-814c-af13db02c3cc" (UID: "4b50a1e4-735a-4467-814c-af13db02c3cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 20:50:09.811257 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.811220    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15523e6f-2ba7-430f-8376-25e1c4db7ae2-kube-api-access-k9lvc" (OuterVolumeSpecName: "kube-api-access-k9lvc") pod "15523e6f-2ba7-430f-8376-25e1c4db7ae2" (UID: "15523e6f-2ba7-430f-8376-25e1c4db7ae2"). InnerVolumeSpecName "kube-api-access-k9lvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:50:09.910431 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.910398    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b50a1e4-735a-4467-814c-af13db02c3cc-openshift-service-ca\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:50:09.910431 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.910426    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9lvc\" (UniqueName: \"kubernetes.io/projected/15523e6f-2ba7-430f-8376-25e1c4db7ae2-kube-api-access-k9lvc\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:50:09.910431 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.910438    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b50a1e4-735a-4467-814c-af13db02c3cc-metrics-certs\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:50:09.910649 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.910448    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b50a1e4-735a-4467-814c-af13db02c3cc-maas-api-tls\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:50:09.910649 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.910457    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72hhf\" (UniqueName: \"kubernetes.io/projected/ccb9ef04-59dc-410d-a463-d6dd13b925dd-kube-api-access-72hhf\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:50:09.910649 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.910465    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15523e6f-2ba7-430f-8376-25e1c4db7ae2-plugins-config-volume\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:50:09.910649 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:09.910475    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slllc\" (UniqueName: \"kubernetes.io/projected/4b50a1e4-735a-4467-814c-af13db02c3cc-kube-api-access-slllc\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:50:10.516408 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:10.516381    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"
Sep 08 20:50:10.516859 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:10.516380    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc" event={"ID":"4b50a1e4-735a-4467-814c-af13db02c3cc","Type":"ContainerDied","Data":"cc1365fc91145fe615b4986044b1151bfcc20d6626b68cd5d2080e0ca528b37f"}
Sep 08 20:50:10.516859 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:10.516514    2552 scope.go:117] "RemoveContainer" containerID="ee97d6d5d33cd171d0f822a9921ae32b805a54118358df697b1b02e543526a74"
Sep 08 20:50:10.517905 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:10.517763    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg"
Sep 08 20:50:10.517905 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:10.517811    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg" event={"ID":"15523e6f-2ba7-430f-8376-25e1c4db7ae2","Type":"ContainerDied","Data":"caf4595f068290b45fc5fedc27dab574f7243d6b7c8384caa422c9fa128447ee"}
Sep 08 20:50:10.519320 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:10.519299    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl"
Sep 08 20:50:10.519467 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:10.519295    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl" event={"ID":"ccb9ef04-59dc-410d-a463-d6dd13b925dd","Type":"ContainerDied","Data":"7ff0e6345c5201d10168763d93dfa76cddcb722904d4e1e6ef9bad7e13aea988"}
Sep 08 20:50:10.526793 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:10.526773    2552 scope.go:117] "RemoveContainer" containerID="b6b5ed7b7dd500d802dff063cfbdb61108c3dd6d3c4f11bb451136553d27cadb"
Sep 08 20:50:10.534835 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:10.534810    2552 scope.go:117] "RemoveContainer" containerID="d765af8be11c38554ca23b12861ecc05e2c53a071b63fddf583651782caab58f"
Sep 08 20:50:10.539573 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:10.539551    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"]
Sep 08 20:50:10.543558 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:10.543539    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-792dc6-78b7f66f9c-lr4tc"]
Sep 08 20:50:10.553216 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:10.553195    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg"]
Sep 08 20:50:10.557221 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:10.557201    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-792dc6-7788f7d446-7hrkg"]
Sep 08 20:50:10.569723 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:10.569666    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl"]
Sep 08 20:50:10.573812 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:10.573791    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-792dc6-769cf8c495-6hsjl"]
Sep 08 20:50:11.230988 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:11.230953    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15523e6f-2ba7-430f-8376-25e1c4db7ae2" path="/var/lib/kubelet/pods/15523e6f-2ba7-430f-8376-25e1c4db7ae2/volumes"
Sep 08 20:50:11.231361 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:11.231348    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b50a1e4-735a-4467-814c-af13db02c3cc" path="/var/lib/kubelet/pods/4b50a1e4-735a-4467-814c-af13db02c3cc/volumes"
Sep 08 20:50:11.231662 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:11.231651    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccb9ef04-59dc-410d-a463-d6dd13b925dd" path="/var/lib/kubelet/pods/ccb9ef04-59dc-410d-a463-d6dd13b925dd/volumes"
Sep 08 20:50:17.000489 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:17.000455    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:50:27.047226 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.047190    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"]
Sep 08 20:50:27.047719 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.047422    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw" podUID="fd1d278e-bc11-43f3-88fa-195158176a34" containerName="maas-api" containerID="cri-o://43b7b9dc7cc92e0075d00c3ae9d0639bd6f9e44e954dee3617df1334125549bf" gracePeriod=30
Sep 08 20:50:27.289495 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.289468    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"
Sep 08 20:50:27.351713 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.351686    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1d278e-bc11-43f3-88fa-195158176a34-maas-api-tls\") pod \"fd1d278e-bc11-43f3-88fa-195158176a34\" (UID: \"fd1d278e-bc11-43f3-88fa-195158176a34\") "
Sep 08 20:50:27.351874 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.351782    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1d278e-bc11-43f3-88fa-195158176a34-openshift-service-ca\") pod \"fd1d278e-bc11-43f3-88fa-195158176a34\" (UID: \"fd1d278e-bc11-43f3-88fa-195158176a34\") "
Sep 08 20:50:27.351874 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.351809    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1d278e-bc11-43f3-88fa-195158176a34-metrics-certs\") pod \"fd1d278e-bc11-43f3-88fa-195158176a34\" (UID: \"fd1d278e-bc11-43f3-88fa-195158176a34\") "
Sep 08 20:50:27.351874 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.351832    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmtsr\" (UniqueName: \"kubernetes.io/projected/fd1d278e-bc11-43f3-88fa-195158176a34-kube-api-access-xmtsr\") pod \"fd1d278e-bc11-43f3-88fa-195158176a34\" (UID: \"fd1d278e-bc11-43f3-88fa-195158176a34\") "
Sep 08 20:50:27.352187 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.352164    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd1d278e-bc11-43f3-88fa-195158176a34-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fd1d278e-bc11-43f3-88fa-195158176a34" (UID: "fd1d278e-bc11-43f3-88fa-195158176a34"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:50:27.353853 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.353823    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd1d278e-bc11-43f3-88fa-195158176a34-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fd1d278e-bc11-43f3-88fa-195158176a34" (UID: "fd1d278e-bc11-43f3-88fa-195158176a34"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 20:50:27.354293 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.354265    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd1d278e-bc11-43f3-88fa-195158176a34-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fd1d278e-bc11-43f3-88fa-195158176a34" (UID: "fd1d278e-bc11-43f3-88fa-195158176a34"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 20:50:27.354397 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.354289    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd1d278e-bc11-43f3-88fa-195158176a34-kube-api-access-xmtsr" (OuterVolumeSpecName: "kube-api-access-xmtsr") pod "fd1d278e-bc11-43f3-88fa-195158176a34" (UID: "fd1d278e-bc11-43f3-88fa-195158176a34"). InnerVolumeSpecName "kube-api-access-xmtsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:50:27.453688 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.453644    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1d278e-bc11-43f3-88fa-195158176a34-maas-api-tls\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:50:27.453688 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.453677    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1d278e-bc11-43f3-88fa-195158176a34-openshift-service-ca\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:50:27.453688 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.453691    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1d278e-bc11-43f3-88fa-195158176a34-metrics-certs\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:50:27.453984 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.453703    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmtsr\" (UniqueName: \"kubernetes.io/projected/fd1d278e-bc11-43f3-88fa-195158176a34-kube-api-access-xmtsr\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:50:27.599487 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.599448    2552 generic.go:356] "Generic (PLEG): container finished" podID="fd1d278e-bc11-43f3-88fa-195158176a34" containerID="43b7b9dc7cc92e0075d00c3ae9d0639bd6f9e44e954dee3617df1334125549bf" exitCode=0
Sep 08 20:50:27.599653 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.599497    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"
Sep 08 20:50:27.599653 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.599532    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw" event={"ID":"fd1d278e-bc11-43f3-88fa-195158176a34","Type":"ContainerDied","Data":"43b7b9dc7cc92e0075d00c3ae9d0639bd6f9e44e954dee3617df1334125549bf"}
Sep 08 20:50:27.599653 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.599587    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw" event={"ID":"fd1d278e-bc11-43f3-88fa-195158176a34","Type":"ContainerDied","Data":"cdb4da567ec9a30036cc6521d85034c0ff97d69bbae5c95d088d163b4d4fb0d7"}
Sep 08 20:50:27.599653 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.599608    2552 scope.go:117] "RemoveContainer" containerID="43b7b9dc7cc92e0075d00c3ae9d0639bd6f9e44e954dee3617df1334125549bf"
Sep 08 20:50:27.609122 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.609102    2552 scope.go:117] "RemoveContainer" containerID="43b7b9dc7cc92e0075d00c3ae9d0639bd6f9e44e954dee3617df1334125549bf"
Sep 08 20:50:27.609407 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:50:27.609389    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43b7b9dc7cc92e0075d00c3ae9d0639bd6f9e44e954dee3617df1334125549bf\": container with ID starting with 43b7b9dc7cc92e0075d00c3ae9d0639bd6f9e44e954dee3617df1334125549bf not found: ID does not exist" containerID="43b7b9dc7cc92e0075d00c3ae9d0639bd6f9e44e954dee3617df1334125549bf"
Sep 08 20:50:27.609494 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.609419    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b7b9dc7cc92e0075d00c3ae9d0639bd6f9e44e954dee3617df1334125549bf"} err="failed to get container status \"43b7b9dc7cc92e0075d00c3ae9d0639bd6f9e44e954dee3617df1334125549bf\": rpc error: code = NotFound desc = could not find container \"43b7b9dc7cc92e0075d00c3ae9d0639bd6f9e44e954dee3617df1334125549bf\": container with ID starting with 43b7b9dc7cc92e0075d00c3ae9d0639bd6f9e44e954dee3617df1334125549bf not found: ID does not exist"
Sep 08 20:50:27.619974 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.619945    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"]
Sep 08 20:50:27.623530 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:27.623510    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27d4c1-84c5f577df-wpcvw"]
Sep 08 20:50:28.691470 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:28.691431    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:50:29.225231 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:29.225188    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd1d278e-bc11-43f3-88fa-195158176a34" path="/var/lib/kubelet/pods/fd1d278e-bc11-43f3-88fa-195158176a34/volumes"
Sep 08 20:50:41.897655 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:41.897622    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:50:44.816377 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:44.816342    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"]
Sep 08 20:50:44.816789 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:44.816602    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf" podUID="090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13" containerName="maas-api" containerID="cri-o://9a877a0e0182fc0da7eed75669ebc4aa756e9e39bf703690b49e42f122b1b31d" gracePeriod=30
Sep 08 20:50:45.056425 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:45.056395    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"
Sep 08 20:50:45.204212 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:45.204176    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2mpt\" (UniqueName: \"kubernetes.io/projected/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-kube-api-access-p2mpt\") pod \"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13\" (UID: \"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13\") "
Sep 08 20:50:45.204395 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:45.204232    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-openshift-service-ca\") pod \"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13\" (UID: \"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13\") "
Sep 08 20:50:45.204395 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:45.204279    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-metrics-certs\") pod \"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13\" (UID: \"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13\") "
Sep 08 20:50:45.204395 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:45.204302    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-maas-api-tls\") pod \"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13\" (UID: \"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13\") "
Sep 08 20:50:45.204648 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:45.204627    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13" (UID: "090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:50:45.206423 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:45.206394    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13" (UID: "090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 20:50:45.206549 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:45.206428    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13" (UID: "090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 20:50:45.206549 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:45.206543    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-kube-api-access-p2mpt" (OuterVolumeSpecName: "kube-api-access-p2mpt") pod "090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13" (UID: "090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13"). InnerVolumeSpecName "kube-api-access-p2mpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:50:45.305721 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:45.305688    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2mpt\" (UniqueName: \"kubernetes.io/projected/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-kube-api-access-p2mpt\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:50:45.305721 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:45.305718    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-openshift-service-ca\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:50:45.305891 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:45.305731    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-metrics-certs\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:50:45.305891 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:45.305742    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13-maas-api-tls\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:50:45.685427 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:45.685389    2552 generic.go:356] "Generic (PLEG): container finished" podID="090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13" containerID="9a877a0e0182fc0da7eed75669ebc4aa756e9e39bf703690b49e42f122b1b31d" exitCode=0
Sep 08 20:50:45.685592 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:45.685434    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"
Sep 08 20:50:45.685592 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:45.685450    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf" event={"ID":"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13","Type":"ContainerDied","Data":"9a877a0e0182fc0da7eed75669ebc4aa756e9e39bf703690b49e42f122b1b31d"}
Sep 08 20:50:45.685592 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:45.685500    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf" event={"ID":"090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13","Type":"ContainerDied","Data":"ba46367695d44daf9caf280248d942fb7f3361ced512f23b2fdb17a6ae4bf574"}
Sep 08 20:50:45.685592 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:45.685522    2552 scope.go:117] "RemoveContainer" containerID="9a877a0e0182fc0da7eed75669ebc4aa756e9e39bf703690b49e42f122b1b31d"
Sep 08 20:50:45.693793 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:45.693774    2552 scope.go:117] "RemoveContainer" containerID="9a877a0e0182fc0da7eed75669ebc4aa756e9e39bf703690b49e42f122b1b31d"
Sep 08 20:50:45.694032 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:50:45.694017    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a877a0e0182fc0da7eed75669ebc4aa756e9e39bf703690b49e42f122b1b31d\": container with ID starting with 9a877a0e0182fc0da7eed75669ebc4aa756e9e39bf703690b49e42f122b1b31d not found: ID does not exist" containerID="9a877a0e0182fc0da7eed75669ebc4aa756e9e39bf703690b49e42f122b1b31d"
Sep 08 20:50:45.694069 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:45.694041    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a877a0e0182fc0da7eed75669ebc4aa756e9e39bf703690b49e42f122b1b31d"} err="failed to get container status \"9a877a0e0182fc0da7eed75669ebc4aa756e9e39bf703690b49e42f122b1b31d\": rpc error: code = NotFound desc = could not find container \"9a877a0e0182fc0da7eed75669ebc4aa756e9e39bf703690b49e42f122b1b31d\": container with ID starting with 9a877a0e0182fc0da7eed75669ebc4aa756e9e39bf703690b49e42f122b1b31d not found: ID does not exist"
Sep 08 20:50:45.701429 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:45.701386    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"]
Sep 08 20:50:45.703974 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:45.703953    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9a1-557c897579-7wwqf"]
Sep 08 20:50:47.223694 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:47.223661    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13" path="/var/lib/kubelet/pods/090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13/volumes"
Sep 08 20:50:58.288282 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:50:58.288229    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:51:08.197166 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.197128    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:51:08.751453 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.751419    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"]
Sep 08 20:51:08.751902 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.751884    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbff241a-9952-44c5-81ea-5b9c0b864e05" containerName="maas-api"
Sep 08 20:51:08.751902 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.751901    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbff241a-9952-44c5-81ea-5b9c0b864e05" containerName="maas-api"
Sep 08 20:51:08.752111 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.751916    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15523e6f-2ba7-430f-8376-25e1c4db7ae2" containerName="payload-pre-processing"
Sep 08 20:51:08.752111 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.751925    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="15523e6f-2ba7-430f-8376-25e1c4db7ae2" containerName="payload-pre-processing"
Sep 08 20:51:08.752111 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.751935    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb9ef04-59dc-410d-a463-d6dd13b925dd" containerName="payload-processing"
Sep 08 20:51:08.752111 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.751943    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb9ef04-59dc-410d-a463-d6dd13b925dd" containerName="payload-processing"
Sep 08 20:51:08.752111 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.751958    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13" containerName="maas-api"
Sep 08 20:51:08.752111 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.751966    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13" containerName="maas-api"
Sep 08 20:51:08.752111 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.751978    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd1d278e-bc11-43f3-88fa-195158176a34" containerName="maas-api"
Sep 08 20:51:08.752111 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.751986    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd1d278e-bc11-43f3-88fa-195158176a34" containerName="maas-api"
Sep 08 20:51:08.752111 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.752006    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b50a1e4-735a-4467-814c-af13db02c3cc" containerName="maas-api"
Sep 08 20:51:08.752111 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.752014    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b50a1e4-735a-4467-814c-af13db02c3cc" containerName="maas-api"
Sep 08 20:51:08.752111 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.752086    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd1d278e-bc11-43f3-88fa-195158176a34" containerName="maas-api"
Sep 08 20:51:08.752111 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.752105    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbff241a-9952-44c5-81ea-5b9c0b864e05" containerName="maas-api"
Sep 08 20:51:08.752111 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.752117    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="15523e6f-2ba7-430f-8376-25e1c4db7ae2" containerName="payload-pre-processing"
Sep 08 20:51:08.752826 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.752126    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb9ef04-59dc-410d-a463-d6dd13b925dd" containerName="payload-processing"
Sep 08 20:51:08.752826 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.752134    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="090e91e2-a6cf-4a3a-9cbf-ab1ea9710e13" containerName="maas-api"
Sep 08 20:51:08.752826 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.752142    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b50a1e4-735a-4467-814c-af13db02c3cc" containerName="maas-api"
Sep 08 20:51:08.755604 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.755580    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"
Sep 08 20:51:08.759122 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.759098    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 20:51:08.759264 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.759160    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-z8244\""
Sep 08 20:51:08.759264 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.759198    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 20:51:08.759264 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.759105    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 20:51:08.759498 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.759098    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-aab326-serving-cert\""
Sep 08 20:51:08.766757 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.766734    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"]
Sep 08 20:51:08.807136 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.807105    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35e17d54-17a8-4714-9694-71ded5de6142-metrics-certs\") pod \"maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s\" (UID: \"35e17d54-17a8-4714-9694-71ded5de6142\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"
Sep 08 20:51:08.807311 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.807146    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfjxf\" (UniqueName: \"kubernetes.io/projected/35e17d54-17a8-4714-9694-71ded5de6142-kube-api-access-jfjxf\") pod \"maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s\" (UID: \"35e17d54-17a8-4714-9694-71ded5de6142\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"
Sep 08 20:51:08.807311 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.807221    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35e17d54-17a8-4714-9694-71ded5de6142-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s\" (UID: \"35e17d54-17a8-4714-9694-71ded5de6142\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"
Sep 08 20:51:08.807430 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.807316    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35e17d54-17a8-4714-9694-71ded5de6142-maas-api-tls\") pod \"maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s\" (UID: \"35e17d54-17a8-4714-9694-71ded5de6142\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"
Sep 08 20:51:08.908388 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.908357    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35e17d54-17a8-4714-9694-71ded5de6142-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s\" (UID: \"35e17d54-17a8-4714-9694-71ded5de6142\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"
Sep 08 20:51:08.908576 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.908422    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35e17d54-17a8-4714-9694-71ded5de6142-maas-api-tls\") pod \"maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s\" (UID: \"35e17d54-17a8-4714-9694-71ded5de6142\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"
Sep 08 20:51:08.908576 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.908501    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35e17d54-17a8-4714-9694-71ded5de6142-metrics-certs\") pod \"maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s\" (UID: \"35e17d54-17a8-4714-9694-71ded5de6142\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"
Sep 08 20:51:08.908576 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.908530    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfjxf\" (UniqueName: \"kubernetes.io/projected/35e17d54-17a8-4714-9694-71ded5de6142-kube-api-access-jfjxf\") pod \"maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s\" (UID: \"35e17d54-17a8-4714-9694-71ded5de6142\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"
Sep 08 20:51:08.909112 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.909081    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35e17d54-17a8-4714-9694-71ded5de6142-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s\" (UID: \"35e17d54-17a8-4714-9694-71ded5de6142\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"
Sep 08 20:51:08.910820 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.910788    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35e17d54-17a8-4714-9694-71ded5de6142-metrics-certs\") pod \"maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s\" (UID: \"35e17d54-17a8-4714-9694-71ded5de6142\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"
Sep 08 20:51:08.910924 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.910849    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35e17d54-17a8-4714-9694-71ded5de6142-maas-api-tls\") pod \"maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s\" (UID: \"35e17d54-17a8-4714-9694-71ded5de6142\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"
Sep 08 20:51:08.917145 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:08.917123    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfjxf\" (UniqueName: \"kubernetes.io/projected/35e17d54-17a8-4714-9694-71ded5de6142-kube-api-access-jfjxf\") pod \"maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s\" (UID: \"35e17d54-17a8-4714-9694-71ded5de6142\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"
Sep 08 20:51:09.070585 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:09.070493    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"
Sep 08 20:51:09.340566 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:09.340541    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"]
Sep 08 20:51:09.342935 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:51:09.342908    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35e17d54_17a8_4714_9694_71ded5de6142.slice/crio-612a041040c9610abb22cf674db19f17612e50c0d04e32dde20cc6b4505ad5b2 WatchSource:0}: Error finding container 612a041040c9610abb22cf674db19f17612e50c0d04e32dde20cc6b4505ad5b2: Status 404 returned error can't find the container with id 612a041040c9610abb22cf674db19f17612e50c0d04e32dde20cc6b4505ad5b2
Sep 08 20:51:09.392938 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:09.392911    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl"]
Sep 08 20:51:09.397384 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:09.397365    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl"
Sep 08 20:51:09.399640 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:09.399621    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-aab326-dockercfg-bj6g8\""
Sep 08 20:51:09.400018 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:09.400002    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-aab326\""
Sep 08 20:51:09.409181 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:09.409159    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl"]
Sep 08 20:51:09.411961 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:09.411937    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h68vx\" (UniqueName: \"kubernetes.io/projected/e824376a-0a4c-44b3-a1e3-f47d5865f910-kube-api-access-h68vx\") pod \"payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl\" (UID: \"e824376a-0a4c-44b3-a1e3-f47d5865f910\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl"
Sep 08 20:51:09.412087 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:09.412050    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e824376a-0a4c-44b3-a1e3-f47d5865f910-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl\" (UID: \"e824376a-0a4c-44b3-a1e3-f47d5865f910\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl"
Sep 08 20:51:09.512408 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:09.512377    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e824376a-0a4c-44b3-a1e3-f47d5865f910-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl\" (UID: \"e824376a-0a4c-44b3-a1e3-f47d5865f910\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl"
Sep 08 20:51:09.512583 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:09.512420    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h68vx\" (UniqueName: \"kubernetes.io/projected/e824376a-0a4c-44b3-a1e3-f47d5865f910-kube-api-access-h68vx\") pod \"payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl\" (UID: \"e824376a-0a4c-44b3-a1e3-f47d5865f910\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl"
Sep 08 20:51:09.513153 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:09.513134    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e824376a-0a4c-44b3-a1e3-f47d5865f910-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl\" (UID: \"e824376a-0a4c-44b3-a1e3-f47d5865f910\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl"
Sep 08 20:51:09.520145 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:09.520119    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h68vx\" (UniqueName: \"kubernetes.io/projected/e824376a-0a4c-44b3-a1e3-f47d5865f910-kube-api-access-h68vx\") pod \"payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl\" (UID: \"e824376a-0a4c-44b3-a1e3-f47d5865f910\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl"
Sep 08 20:51:09.707554 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:09.707474    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl"
Sep 08 20:51:09.788149 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:09.788115    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s" event={"ID":"35e17d54-17a8-4714-9694-71ded5de6142","Type":"ContainerStarted","Data":"612a041040c9610abb22cf674db19f17612e50c0d04e32dde20cc6b4505ad5b2"}
Sep 08 20:51:09.975904 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:09.975820    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl"]
Sep 08 20:51:09.978495 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:51:09.978467    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode824376a_0a4c_44b3_a1e3_f47d5865f910.slice/crio-81f64bb0b7d34db2fb3f5f46526b0672e8035504f0e0a7aa93dd98b08c69dc1c WatchSource:0}: Error finding container 81f64bb0b7d34db2fb3f5f46526b0672e8035504f0e0a7aa93dd98b08c69dc1c: Status 404 returned error can't find the container with id 81f64bb0b7d34db2fb3f5f46526b0672e8035504f0e0a7aa93dd98b08c69dc1c
Sep 08 20:51:10.793831 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:10.793797    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s" event={"ID":"35e17d54-17a8-4714-9694-71ded5de6142","Type":"ContainerStarted","Data":"c8c6a57b21f827bbf11215ecc91778636ce8a5d32517397bef7d955332258f1b"}
Sep 08 20:51:10.794291 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:10.793856    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"
Sep 08 20:51:10.795364 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:10.795340    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl" event={"ID":"e824376a-0a4c-44b3-a1e3-f47d5865f910","Type":"ContainerStarted","Data":"6e782e11f2268bc74e553b254eb960e87e81ee441349a9f7c36a2edfc7d6f0d2"}
Sep 08 20:51:10.795476 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:10.795386    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl" event={"ID":"e824376a-0a4c-44b3-a1e3-f47d5865f910","Type":"ContainerStarted","Data":"81f64bb0b7d34db2fb3f5f46526b0672e8035504f0e0a7aa93dd98b08c69dc1c"}
Sep 08 20:51:10.795476 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:10.795409    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl"
Sep 08 20:51:10.810338 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:10.810294    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s" podStartSLOduration=2.355519909 podStartE2EDuration="2.810278475s" totalImagesPullingTime="454.758566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:51:10.808197603 +0000 UTC m=+1524.059678960" watchObservedRunningTime="2026-09-08 20:51:10.810278475 +0000 UTC m=+1524.061759831"
Sep 08 20:51:10.824752 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:10.824712    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl" podStartSLOduration=1.8246982269999998 podStartE2EDuration="1.824698227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:51:10.822432148 +0000 UTC m=+1524.073913506" watchObservedRunningTime="2026-09-08 20:51:10.824698227 +0000 UTC m=+1524.076179584"
Sep 08 20:51:16.805062 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:16.805029    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"
Sep 08 20:51:21.107995 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:21.107958    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:51:21.801840 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:21.801812    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl"
Sep 08 20:51:44.492870 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:51:44.492837    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:52:16.894098 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:16.894062    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:52:36.604963 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:36.604923    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:52:42.782041 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:42.781956    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl_e824376a-0a4c-44b3-a1e3-f47d5865f910/payload-processing/0.log"
Sep 08 20:52:47.614306 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:47.614257    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"]
Sep 08 20:52:47.617796 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:47.617778    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"
Sep 08 20:52:47.619830 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:47.619810    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-4126a9-serving-cert\""
Sep 08 20:52:47.629601 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:47.629579    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"]
Sep 08 20:52:47.751425 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:47.751386    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/987cb875-7a57-4fe3-8a33-12c5b0f15393-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b\" (UID: \"987cb875-7a57-4fe3-8a33-12c5b0f15393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"
Sep 08 20:52:47.751615 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:47.751432    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbrjq\" (UniqueName: \"kubernetes.io/projected/987cb875-7a57-4fe3-8a33-12c5b0f15393-kube-api-access-wbrjq\") pod \"maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b\" (UID: \"987cb875-7a57-4fe3-8a33-12c5b0f15393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"
Sep 08 20:52:47.751615 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:47.751460    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/987cb875-7a57-4fe3-8a33-12c5b0f15393-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b\" (UID: \"987cb875-7a57-4fe3-8a33-12c5b0f15393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"
Sep 08 20:52:47.751615 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:47.751535    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/987cb875-7a57-4fe3-8a33-12c5b0f15393-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b\" (UID: \"987cb875-7a57-4fe3-8a33-12c5b0f15393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"
Sep 08 20:52:47.852257 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:47.852201    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/987cb875-7a57-4fe3-8a33-12c5b0f15393-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b\" (UID: \"987cb875-7a57-4fe3-8a33-12c5b0f15393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"
Sep 08 20:52:47.852544 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:47.852266    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/987cb875-7a57-4fe3-8a33-12c5b0f15393-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b\" (UID: \"987cb875-7a57-4fe3-8a33-12c5b0f15393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"
Sep 08 20:52:47.852544 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:47.852297    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbrjq\" (UniqueName: \"kubernetes.io/projected/987cb875-7a57-4fe3-8a33-12c5b0f15393-kube-api-access-wbrjq\") pod \"maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b\" (UID: \"987cb875-7a57-4fe3-8a33-12c5b0f15393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"
Sep 08 20:52:47.852544 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:47.852326    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/987cb875-7a57-4fe3-8a33-12c5b0f15393-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b\" (UID: \"987cb875-7a57-4fe3-8a33-12c5b0f15393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"
Sep 08 20:52:47.852544 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:52:47.852361    2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-serving-cert: secret "maas-api-e2e-ipp-cleanup-4126a9-serving-cert" not found
Sep 08 20:52:47.852544 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:52:47.852428    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/987cb875-7a57-4fe3-8a33-12c5b0f15393-maas-api-tls podName:987cb875-7a57-4fe3-8a33-12c5b0f15393 nodeName:}" failed. No retries permitted until 2026-09-08 20:52:48.352411837 +0000 UTC m=+1621.603893172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/987cb875-7a57-4fe3-8a33-12c5b0f15393-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b" (UID: "987cb875-7a57-4fe3-8a33-12c5b0f15393") : secret "maas-api-e2e-ipp-cleanup-4126a9-serving-cert" not found
Sep 08 20:52:47.852910 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:47.852890    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/987cb875-7a57-4fe3-8a33-12c5b0f15393-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b\" (UID: \"987cb875-7a57-4fe3-8a33-12c5b0f15393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"
Sep 08 20:52:47.854749 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:47.854731    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/987cb875-7a57-4fe3-8a33-12c5b0f15393-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b\" (UID: \"987cb875-7a57-4fe3-8a33-12c5b0f15393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"
Sep 08 20:52:47.862800 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:47.862778    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbrjq\" (UniqueName: \"kubernetes.io/projected/987cb875-7a57-4fe3-8a33-12c5b0f15393-kube-api-access-wbrjq\") pod \"maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b\" (UID: \"987cb875-7a57-4fe3-8a33-12c5b0f15393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"
Sep 08 20:52:48.356533 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:48.356501    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm"]
Sep 08 20:52:48.356892 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:48.356870    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/987cb875-7a57-4fe3-8a33-12c5b0f15393-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b\" (UID: \"987cb875-7a57-4fe3-8a33-12c5b0f15393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"
Sep 08 20:52:48.359217 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:48.359186    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/987cb875-7a57-4fe3-8a33-12c5b0f15393-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b\" (UID: \"987cb875-7a57-4fe3-8a33-12c5b0f15393\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"
Sep 08 20:52:48.359574 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:48.359561    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm"
Sep 08 20:52:48.361752 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:48.361732    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-4126a9\""
Sep 08 20:52:48.361890 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:48.361816    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-4126a9-dockercfg-wd6ds\""
Sep 08 20:52:48.372946 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:48.372923    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm"]
Sep 08 20:52:48.457913 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:48.457878    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrh56\" (UniqueName: \"kubernetes.io/projected/b6587427-4d90-4a93-a9c8-45a8fbc5782a-kube-api-access-qrh56\") pod \"payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm\" (UID: \"b6587427-4d90-4a93-a9c8-45a8fbc5782a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm"
Sep 08 20:52:48.458083 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:48.457925    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6587427-4d90-4a93-a9c8-45a8fbc5782a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm\" (UID: \"b6587427-4d90-4a93-a9c8-45a8fbc5782a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm"
Sep 08 20:52:48.527216 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:48.527188    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"
Sep 08 20:52:48.558663 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:48.558627    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6587427-4d90-4a93-a9c8-45a8fbc5782a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm\" (UID: \"b6587427-4d90-4a93-a9c8-45a8fbc5782a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm"
Sep 08 20:52:48.558810 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:48.558792    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrh56\" (UniqueName: \"kubernetes.io/projected/b6587427-4d90-4a93-a9c8-45a8fbc5782a-kube-api-access-qrh56\") pod \"payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm\" (UID: \"b6587427-4d90-4a93-a9c8-45a8fbc5782a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm"
Sep 08 20:52:48.559333 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:48.559312    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6587427-4d90-4a93-a9c8-45a8fbc5782a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm\" (UID: \"b6587427-4d90-4a93-a9c8-45a8fbc5782a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm"
Sep 08 20:52:48.567159 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:48.567134    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrh56\" (UniqueName: \"kubernetes.io/projected/b6587427-4d90-4a93-a9c8-45a8fbc5782a-kube-api-access-qrh56\") pod \"payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm\" (UID: \"b6587427-4d90-4a93-a9c8-45a8fbc5782a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm"
Sep 08 20:52:48.676857 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:48.676776    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm"
Sep 08 20:52:48.798303 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:48.798271    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"]
Sep 08 20:52:48.801585 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:52:48.801555    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987cb875_7a57_4fe3_8a33_12c5b0f15393.slice/crio-7df7a2451ba134637a180e7b04928c2414d096c29c5762bd8d54216a41c5809f WatchSource:0}: Error finding container 7df7a2451ba134637a180e7b04928c2414d096c29c5762bd8d54216a41c5809f: Status 404 returned error can't find the container with id 7df7a2451ba134637a180e7b04928c2414d096c29c5762bd8d54216a41c5809f
Sep 08 20:52:48.947519 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:48.947494    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm"]
Sep 08 20:52:48.949424 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:52:48.949397    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6587427_4d90_4a93_a9c8_45a8fbc5782a.slice/crio-9a0840f6a954f31e4412ee4aafbed389cc7c4c2ed5db9ff4ebf717e39d5ef6ff WatchSource:0}: Error finding container 9a0840f6a954f31e4412ee4aafbed389cc7c4c2ed5db9ff4ebf717e39d5ef6ff: Status 404 returned error can't find the container with id 9a0840f6a954f31e4412ee4aafbed389cc7c4c2ed5db9ff4ebf717e39d5ef6ff
Sep 08 20:52:49.229376 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:49.229348    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm" event={"ID":"b6587427-4d90-4a93-a9c8-45a8fbc5782a","Type":"ContainerStarted","Data":"553d259a757d7c1bd5c9d1b7f7aae6ecc48a6901d98a2f34d6af1bd8299f65ce"}
Sep 08 20:52:49.229495 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:49.229402    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm" event={"ID":"b6587427-4d90-4a93-a9c8-45a8fbc5782a","Type":"ContainerStarted","Data":"9a0840f6a954f31e4412ee4aafbed389cc7c4c2ed5db9ff4ebf717e39d5ef6ff"}
Sep 08 20:52:49.229495 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:49.229431    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm"
Sep 08 20:52:49.230540 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:49.230517    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b" event={"ID":"987cb875-7a57-4fe3-8a33-12c5b0f15393","Type":"ContainerStarted","Data":"7df7a2451ba134637a180e7b04928c2414d096c29c5762bd8d54216a41c5809f"}
Sep 08 20:52:49.249336 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:49.249288    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm" podStartSLOduration=1.249266544 podStartE2EDuration="1.249266544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:52:49.248998029 +0000 UTC m=+1622.500479386" watchObservedRunningTime="2026-09-08 20:52:49.249266544 +0000 UTC m=+1622.500747902"
Sep 08 20:52:50.236394 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:50.236357    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b" event={"ID":"987cb875-7a57-4fe3-8a33-12c5b0f15393","Type":"ContainerStarted","Data":"cb5fae6ef4942760242064a357f90f04e46afec4bf0917d7bcdaefc3cc8cd88f"}
Sep 08 20:52:50.254143 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:50.254098    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b" podStartSLOduration=2.874706988 podStartE2EDuration="3.254082662s" totalImagesPullingTime="379.375674ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:52:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:52:50.252990025 +0000 UTC m=+1623.504471386" watchObservedRunningTime="2026-09-08 20:52:50.254082662 +0000 UTC m=+1623.505564018"
Sep 08 20:52:51.240775 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:51.240748    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"
Sep 08 20:52:56.901022 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:56.900986    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:52:57.249892 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:52:57.249814    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"
Sep 08 20:53:00.238800 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:00.238770    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm"
Sep 08 20:53:18.174857 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.174821    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"]
Sep 08 20:53:18.175322 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.175040    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b" podUID="987cb875-7a57-4fe3-8a33-12c5b0f15393" containerName="maas-api" containerID="cri-o://cb5fae6ef4942760242064a357f90f04e46afec4bf0917d7bcdaefc3cc8cd88f" gracePeriod=30
Sep 08 20:53:18.245583 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.245543    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm"]
Sep 08 20:53:18.246062 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.246007    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm" podUID="b6587427-4d90-4a93-a9c8-45a8fbc5782a" containerName="payload-pre-processing" containerID="cri-o://553d259a757d7c1bd5c9d1b7f7aae6ecc48a6901d98a2f34d6af1bd8299f65ce" gracePeriod=30
Sep 08 20:53:18.360027 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.359994    2552 generic.go:356] "Generic (PLEG): container finished" podID="987cb875-7a57-4fe3-8a33-12c5b0f15393" containerID="cb5fae6ef4942760242064a357f90f04e46afec4bf0917d7bcdaefc3cc8cd88f" exitCode=0
Sep 08 20:53:18.360160 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.360029    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b" event={"ID":"987cb875-7a57-4fe3-8a33-12c5b0f15393","Type":"ContainerDied","Data":"cb5fae6ef4942760242064a357f90f04e46afec4bf0917d7bcdaefc3cc8cd88f"}
Sep 08 20:53:18.498692 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.498665    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"
Sep 08 20:53:18.527644 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.527615    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm"
Sep 08 20:53:18.626498 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.626469    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6587427-4d90-4a93-a9c8-45a8fbc5782a-plugins-config-volume\") pod \"b6587427-4d90-4a93-a9c8-45a8fbc5782a\" (UID: \"b6587427-4d90-4a93-a9c8-45a8fbc5782a\") "
Sep 08 20:53:18.626820 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.626527    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/987cb875-7a57-4fe3-8a33-12c5b0f15393-maas-api-tls\") pod \"987cb875-7a57-4fe3-8a33-12c5b0f15393\" (UID: \"987cb875-7a57-4fe3-8a33-12c5b0f15393\") "
Sep 08 20:53:18.626820 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.626657    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/987cb875-7a57-4fe3-8a33-12c5b0f15393-metrics-certs\") pod \"987cb875-7a57-4fe3-8a33-12c5b0f15393\" (UID: \"987cb875-7a57-4fe3-8a33-12c5b0f15393\") "
Sep 08 20:53:18.626820 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.626708    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrh56\" (UniqueName: \"kubernetes.io/projected/b6587427-4d90-4a93-a9c8-45a8fbc5782a-kube-api-access-qrh56\") pod \"b6587427-4d90-4a93-a9c8-45a8fbc5782a\" (UID: \"b6587427-4d90-4a93-a9c8-45a8fbc5782a\") "
Sep 08 20:53:18.626820 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.626750    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbrjq\" (UniqueName: \"kubernetes.io/projected/987cb875-7a57-4fe3-8a33-12c5b0f15393-kube-api-access-wbrjq\") pod \"987cb875-7a57-4fe3-8a33-12c5b0f15393\" (UID: \"987cb875-7a57-4fe3-8a33-12c5b0f15393\") "
Sep 08 20:53:18.626820 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.626788    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/987cb875-7a57-4fe3-8a33-12c5b0f15393-openshift-service-ca\") pod \"987cb875-7a57-4fe3-8a33-12c5b0f15393\" (UID: \"987cb875-7a57-4fe3-8a33-12c5b0f15393\") "
Sep 08 20:53:18.627069 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.626890    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6587427-4d90-4a93-a9c8-45a8fbc5782a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b6587427-4d90-4a93-a9c8-45a8fbc5782a" (UID: "b6587427-4d90-4a93-a9c8-45a8fbc5782a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:53:18.627256 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.627207    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6587427-4d90-4a93-a9c8-45a8fbc5782a-plugins-config-volume\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:53:18.627390 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.627216    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987cb875-7a57-4fe3-8a33-12c5b0f15393-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "987cb875-7a57-4fe3-8a33-12c5b0f15393" (UID: "987cb875-7a57-4fe3-8a33-12c5b0f15393"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:53:18.629636 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.629591    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/987cb875-7a57-4fe3-8a33-12c5b0f15393-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "987cb875-7a57-4fe3-8a33-12c5b0f15393" (UID: "987cb875-7a57-4fe3-8a33-12c5b0f15393"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 20:53:18.629636 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.629615    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6587427-4d90-4a93-a9c8-45a8fbc5782a-kube-api-access-qrh56" (OuterVolumeSpecName: "kube-api-access-qrh56") pod "b6587427-4d90-4a93-a9c8-45a8fbc5782a" (UID: "b6587427-4d90-4a93-a9c8-45a8fbc5782a"). InnerVolumeSpecName "kube-api-access-qrh56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:53:18.629807 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.629711    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987cb875-7a57-4fe3-8a33-12c5b0f15393-kube-api-access-wbrjq" (OuterVolumeSpecName: "kube-api-access-wbrjq") pod "987cb875-7a57-4fe3-8a33-12c5b0f15393" (UID: "987cb875-7a57-4fe3-8a33-12c5b0f15393"). InnerVolumeSpecName "kube-api-access-wbrjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:53:18.629885 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.629864    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/987cb875-7a57-4fe3-8a33-12c5b0f15393-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "987cb875-7a57-4fe3-8a33-12c5b0f15393" (UID: "987cb875-7a57-4fe3-8a33-12c5b0f15393"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 20:53:18.727708 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.727621    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/987cb875-7a57-4fe3-8a33-12c5b0f15393-maas-api-tls\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:53:18.727708 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.727649    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/987cb875-7a57-4fe3-8a33-12c5b0f15393-metrics-certs\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:53:18.727708 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.727663    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrh56\" (UniqueName: \"kubernetes.io/projected/b6587427-4d90-4a93-a9c8-45a8fbc5782a-kube-api-access-qrh56\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:53:18.727708 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.727677    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbrjq\" (UniqueName: \"kubernetes.io/projected/987cb875-7a57-4fe3-8a33-12c5b0f15393-kube-api-access-wbrjq\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:53:18.727708 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:18.727690    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/987cb875-7a57-4fe3-8a33-12c5b0f15393-openshift-service-ca\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:53:19.365294 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:19.365229    2552 generic.go:356] "Generic (PLEG): container finished" podID="b6587427-4d90-4a93-a9c8-45a8fbc5782a" containerID="553d259a757d7c1bd5c9d1b7f7aae6ecc48a6901d98a2f34d6af1bd8299f65ce" exitCode=0
Sep 08 20:53:19.365294 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:19.365298    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm"
Sep 08 20:53:19.365780 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:19.365310    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm" event={"ID":"b6587427-4d90-4a93-a9c8-45a8fbc5782a","Type":"ContainerDied","Data":"553d259a757d7c1bd5c9d1b7f7aae6ecc48a6901d98a2f34d6af1bd8299f65ce"}
Sep 08 20:53:19.365780 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:19.365359    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm" event={"ID":"b6587427-4d90-4a93-a9c8-45a8fbc5782a","Type":"ContainerDied","Data":"9a0840f6a954f31e4412ee4aafbed389cc7c4c2ed5db9ff4ebf717e39d5ef6ff"}
Sep 08 20:53:19.365780 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:19.365385    2552 scope.go:117] "RemoveContainer" containerID="553d259a757d7c1bd5c9d1b7f7aae6ecc48a6901d98a2f34d6af1bd8299f65ce"
Sep 08 20:53:19.367137 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:19.367043    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b" event={"ID":"987cb875-7a57-4fe3-8a33-12c5b0f15393","Type":"ContainerDied","Data":"7df7a2451ba134637a180e7b04928c2414d096c29c5762bd8d54216a41c5809f"}
Sep 08 20:53:19.367284 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:19.367146    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"
Sep 08 20:53:19.374428 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:19.374408    2552 scope.go:117] "RemoveContainer" containerID="553d259a757d7c1bd5c9d1b7f7aae6ecc48a6901d98a2f34d6af1bd8299f65ce"
Sep 08 20:53:19.374692 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:53:19.374671    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"553d259a757d7c1bd5c9d1b7f7aae6ecc48a6901d98a2f34d6af1bd8299f65ce\": container with ID starting with 553d259a757d7c1bd5c9d1b7f7aae6ecc48a6901d98a2f34d6af1bd8299f65ce not found: ID does not exist" containerID="553d259a757d7c1bd5c9d1b7f7aae6ecc48a6901d98a2f34d6af1bd8299f65ce"
Sep 08 20:53:19.374773 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:19.374703    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"553d259a757d7c1bd5c9d1b7f7aae6ecc48a6901d98a2f34d6af1bd8299f65ce"} err="failed to get container status \"553d259a757d7c1bd5c9d1b7f7aae6ecc48a6901d98a2f34d6af1bd8299f65ce\": rpc error: code = NotFound desc = could not find container \"553d259a757d7c1bd5c9d1b7f7aae6ecc48a6901d98a2f34d6af1bd8299f65ce\": container with ID starting with 553d259a757d7c1bd5c9d1b7f7aae6ecc48a6901d98a2f34d6af1bd8299f65ce not found: ID does not exist"
Sep 08 20:53:19.374773 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:19.374726    2552 scope.go:117] "RemoveContainer" containerID="cb5fae6ef4942760242064a357f90f04e46afec4bf0917d7bcdaefc3cc8cd88f"
Sep 08 20:53:19.384657 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:19.384636    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm"]
Sep 08 20:53:19.388217 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:19.388197    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4126a9-7756c86994-tl5qm"]
Sep 08 20:53:19.400954 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:19.400929    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"]
Sep 08 20:53:19.403335 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:19.403314    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4126a9-869dd476f8-mmt6b"]
Sep 08 20:53:21.224215 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:21.224184    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="987cb875-7a57-4fe3-8a33-12c5b0f15393" path="/var/lib/kubelet/pods/987cb875-7a57-4fe3-8a33-12c5b0f15393/volumes"
Sep 08 20:53:21.224597 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:21.224541    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6587427-4d90-4a93-a9c8-45a8fbc5782a" path="/var/lib/kubelet/pods/b6587427-4d90-4a93-a9c8-45a8fbc5782a/volumes"
Sep 08 20:53:22.784095 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:22.784059    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:53:42.294165 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:42.294130    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:53:44.680930 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:44.680896    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"]
Sep 08 20:53:44.681329 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:44.681169    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s" podUID="35e17d54-17a8-4714-9694-71ded5de6142" containerName="maas-api" containerID="cri-o://c8c6a57b21f827bbf11215ecc91778636ce8a5d32517397bef7d955332258f1b" gracePeriod=30
Sep 08 20:53:44.756557 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:44.756527    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl"]
Sep 08 20:53:44.756818 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:44.756777    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl" podUID="e824376a-0a4c-44b3-a1e3-f47d5865f910" containerName="payload-processing" containerID="cri-o://6e782e11f2268bc74e553b254eb960e87e81ee441349a9f7c36a2edfc7d6f0d2" gracePeriod=30
Sep 08 20:53:44.999054 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:44.999032    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"
Sep 08 20:53:45.027219 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.026972    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl"
Sep 08 20:53:45.068256 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.068217    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h68vx\" (UniqueName: \"kubernetes.io/projected/e824376a-0a4c-44b3-a1e3-f47d5865f910-kube-api-access-h68vx\") pod \"e824376a-0a4c-44b3-a1e3-f47d5865f910\" (UID: \"e824376a-0a4c-44b3-a1e3-f47d5865f910\") "
Sep 08 20:53:45.068414 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.068303    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfjxf\" (UniqueName: \"kubernetes.io/projected/35e17d54-17a8-4714-9694-71ded5de6142-kube-api-access-jfjxf\") pod \"35e17d54-17a8-4714-9694-71ded5de6142\" (UID: \"35e17d54-17a8-4714-9694-71ded5de6142\") "
Sep 08 20:53:45.068414 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.068321    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e824376a-0a4c-44b3-a1e3-f47d5865f910-plugins-config-volume\") pod \"e824376a-0a4c-44b3-a1e3-f47d5865f910\" (UID: \"e824376a-0a4c-44b3-a1e3-f47d5865f910\") "
Sep 08 20:53:45.068414 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.068370    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35e17d54-17a8-4714-9694-71ded5de6142-maas-api-tls\") pod \"35e17d54-17a8-4714-9694-71ded5de6142\" (UID: \"35e17d54-17a8-4714-9694-71ded5de6142\") "
Sep 08 20:53:45.068414 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.068410    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35e17d54-17a8-4714-9694-71ded5de6142-metrics-certs\") pod \"35e17d54-17a8-4714-9694-71ded5de6142\" (UID: \"35e17d54-17a8-4714-9694-71ded5de6142\") "
Sep 08 20:53:45.068597 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.068434    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35e17d54-17a8-4714-9694-71ded5de6142-openshift-service-ca\") pod \"35e17d54-17a8-4714-9694-71ded5de6142\" (UID: \"35e17d54-17a8-4714-9694-71ded5de6142\") "
Sep 08 20:53:45.068773 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.068725    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e824376a-0a4c-44b3-a1e3-f47d5865f910-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e824376a-0a4c-44b3-a1e3-f47d5865f910" (UID: "e824376a-0a4c-44b3-a1e3-f47d5865f910"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:53:45.069016 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.068975    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35e17d54-17a8-4714-9694-71ded5de6142-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "35e17d54-17a8-4714-9694-71ded5de6142" (UID: "35e17d54-17a8-4714-9694-71ded5de6142"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:53:45.070525 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.070501    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e17d54-17a8-4714-9694-71ded5de6142-kube-api-access-jfjxf" (OuterVolumeSpecName: "kube-api-access-jfjxf") pod "35e17d54-17a8-4714-9694-71ded5de6142" (UID: "35e17d54-17a8-4714-9694-71ded5de6142"). InnerVolumeSpecName "kube-api-access-jfjxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:53:45.070632 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.070534    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e824376a-0a4c-44b3-a1e3-f47d5865f910-kube-api-access-h68vx" (OuterVolumeSpecName: "kube-api-access-h68vx") pod "e824376a-0a4c-44b3-a1e3-f47d5865f910" (UID: "e824376a-0a4c-44b3-a1e3-f47d5865f910"). InnerVolumeSpecName "kube-api-access-h68vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:53:45.070632 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.070529    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35e17d54-17a8-4714-9694-71ded5de6142-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "35e17d54-17a8-4714-9694-71ded5de6142" (UID: "35e17d54-17a8-4714-9694-71ded5de6142"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 20:53:45.070891 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.070872    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35e17d54-17a8-4714-9694-71ded5de6142-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "35e17d54-17a8-4714-9694-71ded5de6142" (UID: "35e17d54-17a8-4714-9694-71ded5de6142"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 20:53:45.169273 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.169220    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35e17d54-17a8-4714-9694-71ded5de6142-maas-api-tls\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:53:45.169273 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.169269    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35e17d54-17a8-4714-9694-71ded5de6142-metrics-certs\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:53:45.169273 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.169280    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35e17d54-17a8-4714-9694-71ded5de6142-openshift-service-ca\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:53:45.169494 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.169289    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h68vx\" (UniqueName: \"kubernetes.io/projected/e824376a-0a4c-44b3-a1e3-f47d5865f910-kube-api-access-h68vx\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:53:45.169494 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.169298    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfjxf\" (UniqueName: \"kubernetes.io/projected/35e17d54-17a8-4714-9694-71ded5de6142-kube-api-access-jfjxf\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:53:45.169494 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.169307    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e824376a-0a4c-44b3-a1e3-f47d5865f910-plugins-config-volume\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:53:45.486038 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.485947    2552 generic.go:356] "Generic (PLEG): container finished" podID="e824376a-0a4c-44b3-a1e3-f47d5865f910" containerID="6e782e11f2268bc74e553b254eb960e87e81ee441349a9f7c36a2edfc7d6f0d2" exitCode=0
Sep 08 20:53:45.486038 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.485996    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl"
Sep 08 20:53:45.486289 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.486028    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl" event={"ID":"e824376a-0a4c-44b3-a1e3-f47d5865f910","Type":"ContainerDied","Data":"6e782e11f2268bc74e553b254eb960e87e81ee441349a9f7c36a2edfc7d6f0d2"}
Sep 08 20:53:45.486289 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.486084    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl" event={"ID":"e824376a-0a4c-44b3-a1e3-f47d5865f910","Type":"ContainerDied","Data":"81f64bb0b7d34db2fb3f5f46526b0672e8035504f0e0a7aa93dd98b08c69dc1c"}
Sep 08 20:53:45.486289 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.486107    2552 scope.go:117] "RemoveContainer" containerID="6e782e11f2268bc74e553b254eb960e87e81ee441349a9f7c36a2edfc7d6f0d2"
Sep 08 20:53:45.487512 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.487490    2552 generic.go:356] "Generic (PLEG): container finished" podID="35e17d54-17a8-4714-9694-71ded5de6142" containerID="c8c6a57b21f827bbf11215ecc91778636ce8a5d32517397bef7d955332258f1b" exitCode=0
Sep 08 20:53:45.487609 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.487548    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"
Sep 08 20:53:45.487666 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.487551    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s" event={"ID":"35e17d54-17a8-4714-9694-71ded5de6142","Type":"ContainerDied","Data":"c8c6a57b21f827bbf11215ecc91778636ce8a5d32517397bef7d955332258f1b"}
Sep 08 20:53:45.487718 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.487687    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s" event={"ID":"35e17d54-17a8-4714-9694-71ded5de6142","Type":"ContainerDied","Data":"612a041040c9610abb22cf674db19f17612e50c0d04e32dde20cc6b4505ad5b2"}
Sep 08 20:53:45.494778 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.494761    2552 scope.go:117] "RemoveContainer" containerID="6e782e11f2268bc74e553b254eb960e87e81ee441349a9f7c36a2edfc7d6f0d2"
Sep 08 20:53:45.495029 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:53:45.495009    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e782e11f2268bc74e553b254eb960e87e81ee441349a9f7c36a2edfc7d6f0d2\": container with ID starting with 6e782e11f2268bc74e553b254eb960e87e81ee441349a9f7c36a2edfc7d6f0d2 not found: ID does not exist" containerID="6e782e11f2268bc74e553b254eb960e87e81ee441349a9f7c36a2edfc7d6f0d2"
Sep 08 20:53:45.495077 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.495037    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e782e11f2268bc74e553b254eb960e87e81ee441349a9f7c36a2edfc7d6f0d2"} err="failed to get container status \"6e782e11f2268bc74e553b254eb960e87e81ee441349a9f7c36a2edfc7d6f0d2\": rpc error: code = NotFound desc = could not find container \"6e782e11f2268bc74e553b254eb960e87e81ee441349a9f7c36a2edfc7d6f0d2\": container with ID starting with 6e782e11f2268bc74e553b254eb960e87e81ee441349a9f7c36a2edfc7d6f0d2 not found: ID does not exist"
Sep 08 20:53:45.495077 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.495054    2552 scope.go:117] "RemoveContainer" containerID="c8c6a57b21f827bbf11215ecc91778636ce8a5d32517397bef7d955332258f1b"
Sep 08 20:53:45.504149 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.504133    2552 scope.go:117] "RemoveContainer" containerID="c8c6a57b21f827bbf11215ecc91778636ce8a5d32517397bef7d955332258f1b"
Sep 08 20:53:45.504470 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:53:45.504454    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8c6a57b21f827bbf11215ecc91778636ce8a5d32517397bef7d955332258f1b\": container with ID starting with c8c6a57b21f827bbf11215ecc91778636ce8a5d32517397bef7d955332258f1b not found: ID does not exist" containerID="c8c6a57b21f827bbf11215ecc91778636ce8a5d32517397bef7d955332258f1b"
Sep 08 20:53:45.504542 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.504479    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c6a57b21f827bbf11215ecc91778636ce8a5d32517397bef7d955332258f1b"} err="failed to get container status \"c8c6a57b21f827bbf11215ecc91778636ce8a5d32517397bef7d955332258f1b\": rpc error: code = NotFound desc = could not find container \"c8c6a57b21f827bbf11215ecc91778636ce8a5d32517397bef7d955332258f1b\": container with ID starting with c8c6a57b21f827bbf11215ecc91778636ce8a5d32517397bef7d955332258f1b not found: ID does not exist"
Sep 08 20:53:45.526175 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.526147    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"]
Sep 08 20:53:45.537674 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.537650    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab326-76ccc99dcc-b5r2s"]
Sep 08 20:53:45.564781 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.564758    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl"]
Sep 08 20:53:45.571722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:45.571698    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-aab326-6f4ff94cf5-mvwwl"]
Sep 08 20:53:47.224902 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:47.224866    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35e17d54-17a8-4714-9694-71ded5de6142" path="/var/lib/kubelet/pods/35e17d54-17a8-4714-9694-71ded5de6142/volumes"
Sep 08 20:53:47.225294 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:47.225214    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e824376a-0a4c-44b3-a1e3-f47d5865f910" path="/var/lib/kubelet/pods/e824376a-0a4c-44b3-a1e3-f47d5865f910/volumes"
Sep 08 20:53:55.628692 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:53:55.628657    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:54:34.477441 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:34.477404    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0566ff-62x9t"]
Sep 08 20:54:34.477913 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:34.477719    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6587427-4d90-4a93-a9c8-45a8fbc5782a" containerName="payload-pre-processing"
Sep 08 20:54:34.477913 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:34.477730    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6587427-4d90-4a93-a9c8-45a8fbc5782a" containerName="payload-pre-processing"
Sep 08 20:54:34.477913 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:34.477746    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e824376a-0a4c-44b3-a1e3-f47d5865f910" containerName="payload-processing"
Sep 08 20:54:34.477913 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:34.477752    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e824376a-0a4c-44b3-a1e3-f47d5865f910" containerName="payload-processing"
Sep 08 20:54:34.477913 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:34.477764    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35e17d54-17a8-4714-9694-71ded5de6142" containerName="maas-api"
Sep 08 20:54:34.477913 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:34.477770    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="35e17d54-17a8-4714-9694-71ded5de6142" containerName="maas-api"
Sep 08 20:54:34.477913 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:34.477777    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="987cb875-7a57-4fe3-8a33-12c5b0f15393" containerName="maas-api"
Sep 08 20:54:34.477913 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:34.477783    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="987cb875-7a57-4fe3-8a33-12c5b0f15393" containerName="maas-api"
Sep 08 20:54:34.477913 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:34.477835    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="987cb875-7a57-4fe3-8a33-12c5b0f15393" containerName="maas-api"
Sep 08 20:54:34.477913 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:34.477843    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="35e17d54-17a8-4714-9694-71ded5de6142" containerName="maas-api"
Sep 08 20:54:34.477913 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:34.477850    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6587427-4d90-4a93-a9c8-45a8fbc5782a" containerName="payload-pre-processing"
Sep 08 20:54:34.477913 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:34.477856    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e824376a-0a4c-44b3-a1e3-f47d5865f910" containerName="payload-processing"
Sep 08 20:54:34.480829 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:34.480807    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0566ff-62x9t"
Sep 08 20:54:34.483196 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:34.483175    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2b2qs\""
Sep 08 20:54:34.483684 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:34.483664    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 20:54:34.490722 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:34.490699    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0566ff-62x9t"]
Sep 08 20:54:34.496665 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:34.496644    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e0805ef-4802-47a9-84fc-a1b3b395ce74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0566ff-62x9t\" (UID: \"3e0805ef-4802-47a9-84fc-a1b3b395ce74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0566ff-62x9t"
Sep 08 20:54:34.598153 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:34.598115    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e0805ef-4802-47a9-84fc-a1b3b395ce74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0566ff-62x9t\" (UID: \"3e0805ef-4802-47a9-84fc-a1b3b395ce74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0566ff-62x9t"
Sep 08 20:54:34.598751 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:34.598729    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e0805ef-4802-47a9-84fc-a1b3b395ce74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0566ff-62x9t\" (UID: \"3e0805ef-4802-47a9-84fc-a1b3b395ce74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0566ff-62x9t"
Sep 08 20:54:34.790826 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:34.790733    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0566ff-62x9t"
Sep 08 20:54:35.072850 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:35.072825    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0566ff-62x9t"]
Sep 08 20:54:35.074798 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:54:35.074770    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e0805ef_4802_47a9_84fc_a1b3b395ce74.slice/crio-02d73ec4b5615819de456859fece353bad3a864742581bd013665fde5a9de772 WatchSource:0}: Error finding container 02d73ec4b5615819de456859fece353bad3a864742581bd013665fde5a9de772: Status 404 returned error can't find the container with id 02d73ec4b5615819de456859fece353bad3a864742581bd013665fde5a9de772
Sep 08 20:54:35.698771 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:35.698732    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0566ff-62x9t" event={"ID":"3e0805ef-4802-47a9-84fc-a1b3b395ce74","Type":"ContainerStarted","Data":"c662aa5454014362d7ae68560e7f3ee27378d093a76f0a003a9a3ac03c674529"}
Sep 08 20:54:35.699262 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:35.698784    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0566ff-62x9t" event={"ID":"3e0805ef-4802-47a9-84fc-a1b3b395ce74","Type":"ContainerStarted","Data":"02d73ec4b5615819de456859fece353bad3a864742581bd013665fde5a9de772"}
Sep 08 20:54:35.719229 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:35.719179    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0566ff-62x9t" podStartSLOduration=1.7191631200000002 podStartE2EDuration="1.71916312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:54:35.716164251 +0000 UTC m=+1728.967645609" watchObservedRunningTime="2026-09-08 20:54:35.71916312 +0000 UTC m=+1728.970644476"
Sep 08 20:54:36.704425 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:36.704392    2552 generic.go:356] "Generic (PLEG): container finished" podID="3e0805ef-4802-47a9-84fc-a1b3b395ce74" containerID="c662aa5454014362d7ae68560e7f3ee27378d093a76f0a003a9a3ac03c674529" exitCode=0
Sep 08 20:54:36.704805 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:36.704453    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0566ff-62x9t" event={"ID":"3e0805ef-4802-47a9-84fc-a1b3b395ce74","Type":"ContainerDied","Data":"c662aa5454014362d7ae68560e7f3ee27378d093a76f0a003a9a3ac03c674529"}
Sep 08 20:54:37.838497 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:37.838467    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0566ff-62x9t"
Sep 08 20:54:37.927583 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:37.927551    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e0805ef-4802-47a9-84fc-a1b3b395ce74-maas-api-service-ca\") pod \"3e0805ef-4802-47a9-84fc-a1b3b395ce74\" (UID: \"3e0805ef-4802-47a9-84fc-a1b3b395ce74\") "
Sep 08 20:54:37.927916 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:37.927892    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e0805ef-4802-47a9-84fc-a1b3b395ce74-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3e0805ef-4802-47a9-84fc-a1b3b395ce74" (UID: "3e0805ef-4802-47a9-84fc-a1b3b395ce74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:54:38.028635 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:38.028566    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e0805ef-4802-47a9-84fc-a1b3b395ce74-maas-api-service-ca\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:54:38.714954 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:38.714914    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0566ff-62x9t" event={"ID":"3e0805ef-4802-47a9-84fc-a1b3b395ce74","Type":"ContainerDied","Data":"02d73ec4b5615819de456859fece353bad3a864742581bd013665fde5a9de772"}
Sep 08 20:54:38.714954 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:38.714945    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0566ff-62x9t"
Sep 08 20:54:38.714954 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:38.714965    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02d73ec4b5615819de456859fece353bad3a864742581bd013665fde5a9de772"
Sep 08 20:54:46.137435 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:46.137399    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0566ff-62x9t"]
Sep 08 20:54:46.153008 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:46.152981    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0566ff-62x9t"]
Sep 08 20:54:47.224820 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:47.224787    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e0805ef-4802-47a9-84fc-a1b3b395ce74" path="/var/lib/kubelet/pods/3e0805ef-4802-47a9-84fc-a1b3b395ce74/volumes"
Sep 08 20:54:48.205751 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:48.205718    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:54:53.817886 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:53.817852    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-78dd9f-lsbj4"]
Sep 08 20:54:53.818275 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:53.818163    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e0805ef-4802-47a9-84fc-a1b3b395ce74" containerName="revoke-keys"
Sep 08 20:54:53.818275 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:53.818176    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e0805ef-4802-47a9-84fc-a1b3b395ce74" containerName="revoke-keys"
Sep 08 20:54:53.818275 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:53.818251    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e0805ef-4802-47a9-84fc-a1b3b395ce74" containerName="revoke-keys"
Sep 08 20:54:53.821040 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:53.821021    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-78dd9f-lsbj4"
Sep 08 20:54:53.824179 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:53.824159    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2b2qs\""
Sep 08 20:54:53.824395 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:53.824383    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 20:54:53.833943 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:53.833921    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-78dd9f-lsbj4"]
Sep 08 20:54:53.867231 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:53.867197    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe69e9c-c322-4ae9-a90c-69086caac0ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-78dd9f-lsbj4\" (UID: \"5fe69e9c-c322-4ae9-a90c-69086caac0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-78dd9f-lsbj4"
Sep 08 20:54:53.967772 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:53.967738    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe69e9c-c322-4ae9-a90c-69086caac0ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-78dd9f-lsbj4\" (UID: \"5fe69e9c-c322-4ae9-a90c-69086caac0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-78dd9f-lsbj4"
Sep 08 20:54:53.968406 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:53.968388    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe69e9c-c322-4ae9-a90c-69086caac0ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-78dd9f-lsbj4\" (UID: \"5fe69e9c-c322-4ae9-a90c-69086caac0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-78dd9f-lsbj4"
Sep 08 20:54:54.130540 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:54.130510    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-78dd9f-lsbj4"
Sep 08 20:54:54.417093 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:54.417067    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-78dd9f-lsbj4"]
Sep 08 20:54:54.419047 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:54:54.419018    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe69e9c_c322_4ae9_a90c_69086caac0ce.slice/crio-12bc4afd0abc5a5b7cdf2dcd3734db6700a6ea49bc63a3dc6840e5ef46e73614 WatchSource:0}: Error finding container 12bc4afd0abc5a5b7cdf2dcd3734db6700a6ea49bc63a3dc6840e5ef46e73614: Status 404 returned error can't find the container with id 12bc4afd0abc5a5b7cdf2dcd3734db6700a6ea49bc63a3dc6840e5ef46e73614
Sep 08 20:54:54.785147 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:54.785119    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-78dd9f-lsbj4" event={"ID":"5fe69e9c-c322-4ae9-a90c-69086caac0ce","Type":"ContainerStarted","Data":"80f6261199602b099536665f395893aeba9d51ee135741a1f952b28608fdeb94"}
Sep 08 20:54:54.785308 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:54.785166    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-78dd9f-lsbj4" event={"ID":"5fe69e9c-c322-4ae9-a90c-69086caac0ce","Type":"ContainerStarted","Data":"12bc4afd0abc5a5b7cdf2dcd3734db6700a6ea49bc63a3dc6840e5ef46e73614"}
Sep 08 20:54:54.824883 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:54.824833    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-78dd9f-lsbj4" podStartSLOduration=1.824818928 podStartE2EDuration="1.824818928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 20:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 20:54:54.82435027 +0000 UTC m=+1748.075831647" watchObservedRunningTime="2026-09-08 20:54:54.824818928 +0000 UTC m=+1748.076300284"
Sep 08 20:54:55.790736 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:55.790705    2552 generic.go:356] "Generic (PLEG): container finished" podID="5fe69e9c-c322-4ae9-a90c-69086caac0ce" containerID="80f6261199602b099536665f395893aeba9d51ee135741a1f952b28608fdeb94" exitCode=0
Sep 08 20:54:55.790916 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:55.790783    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-78dd9f-lsbj4" event={"ID":"5fe69e9c-c322-4ae9-a90c-69086caac0ce","Type":"ContainerDied","Data":"80f6261199602b099536665f395893aeba9d51ee135741a1f952b28608fdeb94"}
Sep 08 20:54:56.914611 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:56.914588    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-78dd9f-lsbj4"
Sep 08 20:54:56.991736 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:56.991698    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe69e9c-c322-4ae9-a90c-69086caac0ce-maas-api-service-ca\") pod \"5fe69e9c-c322-4ae9-a90c-69086caac0ce\" (UID: \"5fe69e9c-c322-4ae9-a90c-69086caac0ce\") "
Sep 08 20:54:56.992045 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:56.992023    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe69e9c-c322-4ae9-a90c-69086caac0ce-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5fe69e9c-c322-4ae9-a90c-69086caac0ce" (UID: "5fe69e9c-c322-4ae9-a90c-69086caac0ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:54:57.093189 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:57.093163    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe69e9c-c322-4ae9-a90c-69086caac0ce-maas-api-service-ca\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:54:57.800654 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:57.800624    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-78dd9f-lsbj4" event={"ID":"5fe69e9c-c322-4ae9-a90c-69086caac0ce","Type":"ContainerDied","Data":"12bc4afd0abc5a5b7cdf2dcd3734db6700a6ea49bc63a3dc6840e5ef46e73614"}
Sep 08 20:54:57.800821 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:57.800669    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12bc4afd0abc5a5b7cdf2dcd3734db6700a6ea49bc63a3dc6840e5ef46e73614"
Sep 08 20:54:57.800821 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:54:57.800631    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-78dd9f-lsbj4"
Sep 08 20:55:04.395219 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:04.395106    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-78dd9f-29815005-df88w"]
Sep 08 20:55:04.401438 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:04.401400    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-78dd9f-29815005-df88w"]
Sep 08 20:55:04.424762 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:04.424733    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd"]
Sep 08 20:55:04.424998 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:04.424958    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd" podUID="5a663a5f-8573-4d0e-ae77-084ad099c501" containerName="payload-processing" containerID="cri-o://0699a044b7f7bcf4c4ec86cbdcd0759994625fca8feb53256da2d8200ee36ecf" gracePeriod=30
Sep 08 20:55:04.670921 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:04.670891    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd"
Sep 08 20:55:04.690326 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:04.690300    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-78dd9f-lsbj4"]
Sep 08 20:55:04.698904 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:04.698876    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-78dd9f-lsbj4"]
Sep 08 20:55:04.760099 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:04.760060    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a663a5f-8573-4d0e-ae77-084ad099c501-plugins-config-volume\") pod \"5a663a5f-8573-4d0e-ae77-084ad099c501\" (UID: \"5a663a5f-8573-4d0e-ae77-084ad099c501\") "
Sep 08 20:55:04.760314 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:04.760125    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h82xf\" (UniqueName: \"kubernetes.io/projected/5a663a5f-8573-4d0e-ae77-084ad099c501-kube-api-access-h82xf\") pod \"5a663a5f-8573-4d0e-ae77-084ad099c501\" (UID: \"5a663a5f-8573-4d0e-ae77-084ad099c501\") "
Sep 08 20:55:04.760471 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:04.760447    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a663a5f-8573-4d0e-ae77-084ad099c501-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a663a5f-8573-4d0e-ae77-084ad099c501" (UID: "5a663a5f-8573-4d0e-ae77-084ad099c501"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:55:04.762143 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:04.762123    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a663a5f-8573-4d0e-ae77-084ad099c501-kube-api-access-h82xf" (OuterVolumeSpecName: "kube-api-access-h82xf") pod "5a663a5f-8573-4d0e-ae77-084ad099c501" (UID: "5a663a5f-8573-4d0e-ae77-084ad099c501"). InnerVolumeSpecName "kube-api-access-h82xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:55:04.830322 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:04.830286    2552 generic.go:356] "Generic (PLEG): container finished" podID="5a663a5f-8573-4d0e-ae77-084ad099c501" containerID="0699a044b7f7bcf4c4ec86cbdcd0759994625fca8feb53256da2d8200ee36ecf" exitCode=0
Sep 08 20:55:04.830496 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:04.830330    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd"
Sep 08 20:55:04.830496 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:04.830370    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd" event={"ID":"5a663a5f-8573-4d0e-ae77-084ad099c501","Type":"ContainerDied","Data":"0699a044b7f7bcf4c4ec86cbdcd0759994625fca8feb53256da2d8200ee36ecf"}
Sep 08 20:55:04.830496 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:04.830420    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd" event={"ID":"5a663a5f-8573-4d0e-ae77-084ad099c501","Type":"ContainerDied","Data":"32ced29ba77ff291469091fa2d6507006b208cb26202d4cde29fd274c3e2e17e"}
Sep 08 20:55:04.830496 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:04.830443    2552 scope.go:117] "RemoveContainer" containerID="0699a044b7f7bcf4c4ec86cbdcd0759994625fca8feb53256da2d8200ee36ecf"
Sep 08 20:55:04.838965 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:04.838949    2552 scope.go:117] "RemoveContainer" containerID="0699a044b7f7bcf4c4ec86cbdcd0759994625fca8feb53256da2d8200ee36ecf"
Sep 08 20:55:04.839228 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:55:04.839204    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0699a044b7f7bcf4c4ec86cbdcd0759994625fca8feb53256da2d8200ee36ecf\": container with ID starting with 0699a044b7f7bcf4c4ec86cbdcd0759994625fca8feb53256da2d8200ee36ecf not found: ID does not exist" containerID="0699a044b7f7bcf4c4ec86cbdcd0759994625fca8feb53256da2d8200ee36ecf"
Sep 08 20:55:04.839358 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:04.839266    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0699a044b7f7bcf4c4ec86cbdcd0759994625fca8feb53256da2d8200ee36ecf"} err="failed to get container status \"0699a044b7f7bcf4c4ec86cbdcd0759994625fca8feb53256da2d8200ee36ecf\": rpc error: code = NotFound desc = could not find container \"0699a044b7f7bcf4c4ec86cbdcd0759994625fca8feb53256da2d8200ee36ecf\": container with ID starting with 0699a044b7f7bcf4c4ec86cbdcd0759994625fca8feb53256da2d8200ee36ecf not found: ID does not exist"
Sep 08 20:55:04.861235 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:04.861212    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a663a5f-8573-4d0e-ae77-084ad099c501-plugins-config-volume\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:55:04.861235 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:04.861233    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h82xf\" (UniqueName: \"kubernetes.io/projected/5a663a5f-8573-4d0e-ae77-084ad099c501-kube-api-access-h82xf\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:55:04.880611 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:04.880579    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd"]
Sep 08 20:55:04.887092 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:04.887068    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-78dd9f-9ff59d967-5pzbd"]
Sep 08 20:55:05.225275 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:05.225223    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a663a5f-8573-4d0e-ae77-084ad099c501" path="/var/lib/kubelet/pods/5a663a5f-8573-4d0e-ae77-084ad099c501/volumes"
Sep 08 20:55:05.225582 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:05.225568    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fe69e9c-c322-4ae9-a90c-69086caac0ce" path="/var/lib/kubelet/pods/5fe69e9c-c322-4ae9-a90c-69086caac0ce/volumes"
Sep 08 20:55:05.225808 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:05.225796    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aedcc132-20b4-4685-a5eb-e6c54772c8a6" path="/var/lib/kubelet/pods/aedcc132-20b4-4685-a5eb-e6c54772c8a6/volumes"
Sep 08 20:55:07.392495 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:07.392462    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:55:12.577526 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:12.577491    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c9590b-87wts"]
Sep 08 20:55:12.577922 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:12.577813    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe69e9c-c322-4ae9-a90c-69086caac0ce" containerName="revoke-keys"
Sep 08 20:55:12.577922 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:12.577823    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe69e9c-c322-4ae9-a90c-69086caac0ce" containerName="revoke-keys"
Sep 08 20:55:12.577922 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:12.577844    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a663a5f-8573-4d0e-ae77-084ad099c501" containerName="payload-processing"
Sep 08 20:55:12.577922 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:12.577850    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a663a5f-8573-4d0e-ae77-084ad099c501" containerName="payload-processing"
Sep 08 20:55:12.577922 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:12.577894    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe69e9c-c322-4ae9-a90c-69086caac0ce" containerName="revoke-keys"
Sep 08 20:55:12.577922 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:12.577902    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a663a5f-8573-4d0e-ae77-084ad099c501" containerName="payload-processing"
Sep 08 20:55:12.582185 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:12.582166    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c9590b-87wts"
Sep 08 20:55:12.584833 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:12.584802    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2b2qs\""
Sep 08 20:55:12.584960 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:12.584854    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 20:55:12.595513 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:12.595491    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c9590b-87wts"]
Sep 08 20:55:12.732942 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:12.732909    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56929e2a-80ab-4a76-8f49-e940b30e4d04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c9590b-87wts\" (UID: \"56929e2a-80ab-4a76-8f49-e940b30e4d04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c9590b-87wts"
Sep 08 20:55:12.833841 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:12.833811    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56929e2a-80ab-4a76-8f49-e940b30e4d04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c9590b-87wts\" (UID: \"56929e2a-80ab-4a76-8f49-e940b30e4d04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c9590b-87wts"
Sep 08 20:55:12.834502 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:12.834479    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56929e2a-80ab-4a76-8f49-e940b30e4d04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c9590b-87wts\" (UID: \"56929e2a-80ab-4a76-8f49-e940b30e4d04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c9590b-87wts"
Sep 08 20:55:12.892103 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:12.892069    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c9590b-87wts"
Sep 08 20:55:13.170321 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:13.170227    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c9590b-87wts"]
Sep 08 20:55:13.173558 ip-10-0-129-118 kubenswrapper[2552]: W0908 20:55:13.173523    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56929e2a_80ab_4a76_8f49_e940b30e4d04.slice/crio-6b481da1c8d34bd9d4fb45056ff46ba543a2ecf73458525bc506d346d3e9a26a WatchSource:0}: Error finding container 6b481da1c8d34bd9d4fb45056ff46ba543a2ecf73458525bc506d346d3e9a26a: Status 404 returned error can't find the container with id 6b481da1c8d34bd9d4fb45056ff46ba543a2ecf73458525bc506d346d3e9a26a
Sep 08 20:55:13.872138 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:13.872109    2552 generic.go:356] "Generic (PLEG): container finished" podID="56929e2a-80ab-4a76-8f49-e940b30e4d04" containerID="20cce19ca67f54191730ee757989d4f6421427042e5066f87cf5ce9b3d2fd8f2" exitCode=0
Sep 08 20:55:13.872520 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:13.872183    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c9590b-87wts" event={"ID":"56929e2a-80ab-4a76-8f49-e940b30e4d04","Type":"ContainerDied","Data":"20cce19ca67f54191730ee757989d4f6421427042e5066f87cf5ce9b3d2fd8f2"}
Sep 08 20:55:13.872520 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:13.872219    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c9590b-87wts" event={"ID":"56929e2a-80ab-4a76-8f49-e940b30e4d04","Type":"ContainerStarted","Data":"6b481da1c8d34bd9d4fb45056ff46ba543a2ecf73458525bc506d346d3e9a26a"}
Sep 08 20:55:15.002771 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:15.002748    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c9590b-87wts"
Sep 08 20:55:15.151348 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:15.151262    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56929e2a-80ab-4a76-8f49-e940b30e4d04-maas-api-service-ca\") pod \"56929e2a-80ab-4a76-8f49-e940b30e4d04\" (UID: \"56929e2a-80ab-4a76-8f49-e940b30e4d04\") "
Sep 08 20:55:15.151605 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:15.151579    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56929e2a-80ab-4a76-8f49-e940b30e4d04-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "56929e2a-80ab-4a76-8f49-e940b30e4d04" (UID: "56929e2a-80ab-4a76-8f49-e940b30e4d04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:55:15.252588 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:15.252556    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56929e2a-80ab-4a76-8f49-e940b30e4d04-maas-api-service-ca\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:55:15.884362 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:15.884327    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c9590b-87wts" event={"ID":"56929e2a-80ab-4a76-8f49-e940b30e4d04","Type":"ContainerDied","Data":"6b481da1c8d34bd9d4fb45056ff46ba543a2ecf73458525bc506d346d3e9a26a"}
Sep 08 20:55:15.884362 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:15.884357    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c9590b-87wts"
Sep 08 20:55:15.884569 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:15.884402    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b481da1c8d34bd9d4fb45056ff46ba543a2ecf73458525bc506d346d3e9a26a"
Sep 08 20:55:23.293805 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:23.293767    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr"]
Sep 08 20:55:23.294211 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:23.294055    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr" podUID="4c505130-34b0-4a53-81b4-fd0c9efeb586" containerName="payload-pre-processing" containerID="cri-o://c040e47fbaf753570db145b59a9dcc06e2e0d231dfad21e49a02733091cd640e" gracePeriod=30
Sep 08 20:55:23.503797 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:23.503766    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c9590b-87wts"]
Sep 08 20:55:23.509772 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:23.509744    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c9590b-87wts"]
Sep 08 20:55:23.548557 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:23.548508    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr"
Sep 08 20:55:23.628396 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:23.628361    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c505130-34b0-4a53-81b4-fd0c9efeb586-plugins-config-volume\") pod \"4c505130-34b0-4a53-81b4-fd0c9efeb586\" (UID: \"4c505130-34b0-4a53-81b4-fd0c9efeb586\") "
Sep 08 20:55:23.628555 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:23.628433    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxdgk\" (UniqueName: \"kubernetes.io/projected/4c505130-34b0-4a53-81b4-fd0c9efeb586-kube-api-access-nxdgk\") pod \"4c505130-34b0-4a53-81b4-fd0c9efeb586\" (UID: \"4c505130-34b0-4a53-81b4-fd0c9efeb586\") "
Sep 08 20:55:23.628762 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:23.628735    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c505130-34b0-4a53-81b4-fd0c9efeb586-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4c505130-34b0-4a53-81b4-fd0c9efeb586" (UID: "4c505130-34b0-4a53-81b4-fd0c9efeb586"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 20:55:23.630472 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:23.630450    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c505130-34b0-4a53-81b4-fd0c9efeb586-kube-api-access-nxdgk" (OuterVolumeSpecName: "kube-api-access-nxdgk") pod "4c505130-34b0-4a53-81b4-fd0c9efeb586" (UID: "4c505130-34b0-4a53-81b4-fd0c9efeb586"). InnerVolumeSpecName "kube-api-access-nxdgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 20:55:23.729167 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:23.729116    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c505130-34b0-4a53-81b4-fd0c9efeb586-plugins-config-volume\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:55:23.729167 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:23.729157    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxdgk\" (UniqueName: \"kubernetes.io/projected/4c505130-34b0-4a53-81b4-fd0c9efeb586-kube-api-access-nxdgk\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 20:55:23.918750 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:23.918711    2552 generic.go:356] "Generic (PLEG): container finished" podID="4c505130-34b0-4a53-81b4-fd0c9efeb586" containerID="c040e47fbaf753570db145b59a9dcc06e2e0d231dfad21e49a02733091cd640e" exitCode=0
Sep 08 20:55:23.918750 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:23.918750    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr" event={"ID":"4c505130-34b0-4a53-81b4-fd0c9efeb586","Type":"ContainerDied","Data":"c040e47fbaf753570db145b59a9dcc06e2e0d231dfad21e49a02733091cd640e"}
Sep 08 20:55:23.918974 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:23.918762    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr"
Sep 08 20:55:23.918974 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:23.918787    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr" event={"ID":"4c505130-34b0-4a53-81b4-fd0c9efeb586","Type":"ContainerDied","Data":"5af4f3062806f4ad095c3ec7970b4bb0db60f4bbb79e16434432a4c61b45a5da"}
Sep 08 20:55:23.918974 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:23.918808    2552 scope.go:117] "RemoveContainer" containerID="c040e47fbaf753570db145b59a9dcc06e2e0d231dfad21e49a02733091cd640e"
Sep 08 20:55:23.927228 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:23.927204    2552 scope.go:117] "RemoveContainer" containerID="c040e47fbaf753570db145b59a9dcc06e2e0d231dfad21e49a02733091cd640e"
Sep 08 20:55:23.927531 ip-10-0-129-118 kubenswrapper[2552]: E0908 20:55:23.927512    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c040e47fbaf753570db145b59a9dcc06e2e0d231dfad21e49a02733091cd640e\": container with ID starting with c040e47fbaf753570db145b59a9dcc06e2e0d231dfad21e49a02733091cd640e not found: ID does not exist" containerID="c040e47fbaf753570db145b59a9dcc06e2e0d231dfad21e49a02733091cd640e"
Sep 08 20:55:23.927579 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:23.927540    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c040e47fbaf753570db145b59a9dcc06e2e0d231dfad21e49a02733091cd640e"} err="failed to get container status \"c040e47fbaf753570db145b59a9dcc06e2e0d231dfad21e49a02733091cd640e\": rpc error: code = NotFound desc = could not find container \"c040e47fbaf753570db145b59a9dcc06e2e0d231dfad21e49a02733091cd640e\": container with ID starting with c040e47fbaf753570db145b59a9dcc06e2e0d231dfad21e49a02733091cd640e not found: ID does not exist"
Sep 08 20:55:23.958053 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:23.958024    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr"]
Sep 08 20:55:23.966589 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:23.966560    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c9590b-9696b7654-vzmpr"]
Sep 08 20:55:25.225222 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:25.225143    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c505130-34b0-4a53-81b4-fd0c9efeb586" path="/var/lib/kubelet/pods/4c505130-34b0-4a53-81b4-fd0c9efeb586/volumes"
Sep 08 20:55:25.225599 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:25.225500    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56929e2a-80ab-4a76-8f49-e940b30e4d04" path="/var/lib/kubelet/pods/56929e2a-80ab-4a76-8f49-e940b30e4d04/volumes"
Sep 08 20:55:25.712615 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:25.712584    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:55:35.390175 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:35.390138    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:55:47.392539 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:55:47.392504    2552 scope.go:117] "RemoveContainer" containerID="4f44221c280bf392b8ac789cef2790646eb86f5f1f1f565b5097c27cc32a1a65"
Sep 08 20:56:59.094115 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:56:59.094075    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:57:24.092392 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:57:24.092350    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:57:31.892381 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:57:31.892347    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:57:39.395929 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:57:39.395892    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:57:46.794179 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:57:46.794144    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:58:01.088793 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:58:01.088753    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:58:09.515683 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:58:09.515652    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:59:37.526252 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:59:37.526189    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:59:43.306160 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:59:43.306123    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:59:52.006821 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:59:52.006787    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 20:59:58.332652 ip-10-0-129-118 kubenswrapper[2552]: I0908 20:59:58.332612    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 21:00:17.922727 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:17.922693    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 21:00:35.194288 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:35.194235    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 21:00:42.824405 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:42.824368    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 21:00:47.445281 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:47.445232    2552 scope.go:117] "RemoveContainer" containerID="c662aa5454014362d7ae68560e7f3ee27378d093a76f0a003a9a3ac03c674529"
Sep 08 21:00:50.281944 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:50.281906    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 21:00:50.848881 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:50.848845    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"]
Sep 08 21:00:50.849403 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:50.849383    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56929e2a-80ab-4a76-8f49-e940b30e4d04" containerName="revoke-keys"
Sep 08 21:00:50.849478 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:50.849405    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="56929e2a-80ab-4a76-8f49-e940b30e4d04" containerName="revoke-keys"
Sep 08 21:00:50.849478 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:50.849435    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c505130-34b0-4a53-81b4-fd0c9efeb586" containerName="payload-pre-processing"
Sep 08 21:00:50.849478 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:50.849442    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c505130-34b0-4a53-81b4-fd0c9efeb586" containerName="payload-pre-processing"
Sep 08 21:00:50.849575 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:50.849531    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="56929e2a-80ab-4a76-8f49-e940b30e4d04" containerName="revoke-keys"
Sep 08 21:00:50.849575 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:50.849545    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c505130-34b0-4a53-81b4-fd0c9efeb586" containerName="payload-pre-processing"
Sep 08 21:00:50.854221 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:50.854198    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"
Sep 08 21:00:50.869486 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:50.869461    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 21:00:50.869486 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:50.869481    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 21:00:50.869678 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:50.869532    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-z8244\""
Sep 08 21:00:50.869678 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:50.869542    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 08 21:00:50.869678 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:50.869563    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 21:00:50.890817 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:50.890795    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"]
Sep 08 21:00:51.019570 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:51.019533    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-openshift-service-ca\") pod \"maas-api-6b96cdc7c6-6hrms\" (UID: \"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c\") " pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"
Sep 08 21:00:51.019570 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:51.019572    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-metrics-certs\") pod \"maas-api-6b96cdc7c6-6hrms\" (UID: \"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c\") " pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"
Sep 08 21:00:51.019809 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:51.019604    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-455s4\" (UniqueName: \"kubernetes.io/projected/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-kube-api-access-455s4\") pod \"maas-api-6b96cdc7c6-6hrms\" (UID: \"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c\") " pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"
Sep 08 21:00:51.019809 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:51.019670    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-maas-api-tls\") pod \"maas-api-6b96cdc7c6-6hrms\" (UID: \"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c\") " pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"
Sep 08 21:00:51.121036 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:51.120932    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-maas-api-tls\") pod \"maas-api-6b96cdc7c6-6hrms\" (UID: \"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c\") " pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"
Sep 08 21:00:51.121036 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:51.121037    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-openshift-service-ca\") pod \"maas-api-6b96cdc7c6-6hrms\" (UID: \"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c\") " pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"
Sep 08 21:00:51.121297 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:51.121053    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-metrics-certs\") pod \"maas-api-6b96cdc7c6-6hrms\" (UID: \"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c\") " pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"
Sep 08 21:00:51.121297 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:51.121071    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-455s4\" (UniqueName: \"kubernetes.io/projected/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-kube-api-access-455s4\") pod \"maas-api-6b96cdc7c6-6hrms\" (UID: \"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c\") " pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"
Sep 08 21:00:51.121737 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:51.121713    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-openshift-service-ca\") pod \"maas-api-6b96cdc7c6-6hrms\" (UID: \"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c\") " pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"
Sep 08 21:00:51.123665 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:51.123636    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-metrics-certs\") pod \"maas-api-6b96cdc7c6-6hrms\" (UID: \"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c\") " pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"
Sep 08 21:00:51.123774 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:51.123636    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-maas-api-tls\") pod \"maas-api-6b96cdc7c6-6hrms\" (UID: \"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c\") " pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"
Sep 08 21:00:51.133845 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:51.133824    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-455s4\" (UniqueName: \"kubernetes.io/projected/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-kube-api-access-455s4\") pod \"maas-api-6b96cdc7c6-6hrms\" (UID: \"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c\") " pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"
Sep 08 21:00:51.164520 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:51.164498    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"
Sep 08 21:00:51.453169 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:51.453087    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"]
Sep 08 21:00:51.458943 ip-10-0-129-118 kubenswrapper[2552]: W0908 21:00:51.458918    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4bd910f_2aa2_45cc_9f56_d4d98392ac4c.slice/crio-f971fefcf09ea888b98e26feef2e6714aeb37e195f4e5d3207cbcc2ccc92df0c WatchSource:0}: Error finding container f971fefcf09ea888b98e26feef2e6714aeb37e195f4e5d3207cbcc2ccc92df0c: Status 404 returned error can't find the container with id f971fefcf09ea888b98e26feef2e6714aeb37e195f4e5d3207cbcc2ccc92df0c
Sep 08 21:00:51.460215 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:51.460199    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 21:00:52.300746 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:52.300710    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms" event={"ID":"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c","Type":"ContainerStarted","Data":"4f63d1ac5164d15b87795eca94742f593f7b0078630ab4ae4ec50836e8c5c1a7"}
Sep 08 21:00:52.300916 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:52.300766    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms" event={"ID":"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c","Type":"ContainerStarted","Data":"f971fefcf09ea888b98e26feef2e6714aeb37e195f4e5d3207cbcc2ccc92df0c"}
Sep 08 21:00:52.300916 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:52.300806    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"
Sep 08 21:00:52.320891 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:52.320841    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms" podStartSLOduration=1.6630253069999998 podStartE2EDuration="2.320825919s" totalImagesPullingTime="657.800612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 21:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 21:00:52.319692119 +0000 UTC m=+2105.571173476" watchObservedRunningTime="2026-09-08 21:00:52.320825919 +0000 UTC m=+2105.572307273"
Sep 08 21:00:58.311185 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:58.311155    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"
Sep 08 21:00:58.525164 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:58.525128    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 21:00:58.593847 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:58.593812    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b8bc4d5d4-hvtll"]
Sep 08 21:00:58.597248 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:58.597224    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b8bc4d5d4-hvtll"
Sep 08 21:00:58.608383 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:58.608360    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b8bc4d5d4-hvtll"]
Sep 08 21:00:58.686882 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:58.686848    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e5ea18d-12a7-4285-99d0-5a0fd3477b30-metrics-certs\") pod \"maas-api-6b8bc4d5d4-hvtll\" (UID: \"3e5ea18d-12a7-4285-99d0-5a0fd3477b30\") " pod="odh-ai-gateway-infra/maas-api-6b8bc4d5d4-hvtll"
Sep 08 21:00:58.687033 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:58.686989    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e5ea18d-12a7-4285-99d0-5a0fd3477b30-openshift-service-ca\") pod \"maas-api-6b8bc4d5d4-hvtll\" (UID: \"3e5ea18d-12a7-4285-99d0-5a0fd3477b30\") " pod="odh-ai-gateway-infra/maas-api-6b8bc4d5d4-hvtll"
Sep 08 21:00:58.687033 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:58.687018    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p48zh\" (UniqueName: \"kubernetes.io/projected/3e5ea18d-12a7-4285-99d0-5a0fd3477b30-kube-api-access-p48zh\") pod \"maas-api-6b8bc4d5d4-hvtll\" (UID: \"3e5ea18d-12a7-4285-99d0-5a0fd3477b30\") " pod="odh-ai-gateway-infra/maas-api-6b8bc4d5d4-hvtll"
Sep 08 21:00:58.687139 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:58.687067    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e5ea18d-12a7-4285-99d0-5a0fd3477b30-maas-api-tls\") pod \"maas-api-6b8bc4d5d4-hvtll\" (UID: \"3e5ea18d-12a7-4285-99d0-5a0fd3477b30\") " pod="odh-ai-gateway-infra/maas-api-6b8bc4d5d4-hvtll"
Sep 08 21:00:58.788058 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:58.788024    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e5ea18d-12a7-4285-99d0-5a0fd3477b30-metrics-certs\") pod \"maas-api-6b8bc4d5d4-hvtll\" (UID: \"3e5ea18d-12a7-4285-99d0-5a0fd3477b30\") " pod="odh-ai-gateway-infra/maas-api-6b8bc4d5d4-hvtll"
Sep 08 21:00:58.788268 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:58.788069    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e5ea18d-12a7-4285-99d0-5a0fd3477b30-openshift-service-ca\") pod \"maas-api-6b8bc4d5d4-hvtll\" (UID: \"3e5ea18d-12a7-4285-99d0-5a0fd3477b30\") " pod="odh-ai-gateway-infra/maas-api-6b8bc4d5d4-hvtll"
Sep 08 21:00:58.788268 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:58.788092    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48zh\" (UniqueName: \"kubernetes.io/projected/3e5ea18d-12a7-4285-99d0-5a0fd3477b30-kube-api-access-p48zh\") pod \"maas-api-6b8bc4d5d4-hvtll\" (UID: \"3e5ea18d-12a7-4285-99d0-5a0fd3477b30\") " pod="odh-ai-gateway-infra/maas-api-6b8bc4d5d4-hvtll"
Sep 08 21:00:58.788268 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:58.788207    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e5ea18d-12a7-4285-99d0-5a0fd3477b30-maas-api-tls\") pod \"maas-api-6b8bc4d5d4-hvtll\" (UID: \"3e5ea18d-12a7-4285-99d0-5a0fd3477b30\") " pod="odh-ai-gateway-infra/maas-api-6b8bc4d5d4-hvtll"
Sep 08 21:00:58.788756 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:58.788730    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e5ea18d-12a7-4285-99d0-5a0fd3477b30-openshift-service-ca\") pod \"maas-api-6b8bc4d5d4-hvtll\" (UID: \"3e5ea18d-12a7-4285-99d0-5a0fd3477b30\") " pod="odh-ai-gateway-infra/maas-api-6b8bc4d5d4-hvtll"
Sep 08 21:00:58.790608 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:58.790588    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e5ea18d-12a7-4285-99d0-5a0fd3477b30-maas-api-tls\") pod \"maas-api-6b8bc4d5d4-hvtll\" (UID: \"3e5ea18d-12a7-4285-99d0-5a0fd3477b30\") " pod="odh-ai-gateway-infra/maas-api-6b8bc4d5d4-hvtll"
Sep 08 21:00:58.790691 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:58.790590    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e5ea18d-12a7-4285-99d0-5a0fd3477b30-metrics-certs\") pod \"maas-api-6b8bc4d5d4-hvtll\" (UID: \"3e5ea18d-12a7-4285-99d0-5a0fd3477b30\") " pod="odh-ai-gateway-infra/maas-api-6b8bc4d5d4-hvtll"
Sep 08 21:00:58.797022 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:58.797003    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p48zh\" (UniqueName: \"kubernetes.io/projected/3e5ea18d-12a7-4285-99d0-5a0fd3477b30-kube-api-access-p48zh\") pod \"maas-api-6b8bc4d5d4-hvtll\" (UID: \"3e5ea18d-12a7-4285-99d0-5a0fd3477b30\") " pod="odh-ai-gateway-infra/maas-api-6b8bc4d5d4-hvtll"
Sep 08 21:00:58.907220 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:58.907164    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b8bc4d5d4-hvtll"
Sep 08 21:00:59.192721 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:59.192694    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b8bc4d5d4-hvtll"]
Sep 08 21:00:59.194438 ip-10-0-129-118 kubenswrapper[2552]: W0908 21:00:59.194412    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5ea18d_12a7_4285_99d0_5a0fd3477b30.slice/crio-dfa6b2f7f2abb6f996b63c88fb871733a546df761ab1d6c824f9889009272cec WatchSource:0}: Error finding container dfa6b2f7f2abb6f996b63c88fb871733a546df761ab1d6c824f9889009272cec: Status 404 returned error can't find the container with id dfa6b2f7f2abb6f996b63c88fb871733a546df761ab1d6c824f9889009272cec
Sep 08 21:00:59.330029 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:00:59.329996    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b8bc4d5d4-hvtll" event={"ID":"3e5ea18d-12a7-4285-99d0-5a0fd3477b30","Type":"ContainerStarted","Data":"dfa6b2f7f2abb6f996b63c88fb871733a546df761ab1d6c824f9889009272cec"}
Sep 08 21:01:00.336019 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:00.335982    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b8bc4d5d4-hvtll" event={"ID":"3e5ea18d-12a7-4285-99d0-5a0fd3477b30","Type":"ContainerStarted","Data":"cad4339c082414d109b6e7fa1a00c856f799597854c13314d83c3ecefeb82a3a"}
Sep 08 21:01:00.336564 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:00.336126    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b8bc4d5d4-hvtll"
Sep 08 21:01:05.508854 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:05.508791    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b8bc4d5d4-hvtll" podStartSLOduration=7.183626619 podStartE2EDuration="7.508773697s" totalImagesPullingTime="325.147078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 21:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 21:01:00.356384698 +0000 UTC m=+2113.607866055" watchObservedRunningTime="2026-09-08 21:01:05.508773697 +0000 UTC m=+2118.760255089"
Sep 08 21:01:05.510003 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:05.509983    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 21:01:06.346275 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:06.346229    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b8bc4d5d4-hvtll"
Sep 08 21:01:06.401289 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:06.401254    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"]
Sep 08 21:01:06.401509 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:06.401466    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms" podUID="a4bd910f-2aa2-45cc-9f56-d4d98392ac4c" containerName="maas-api" containerID="cri-o://4f63d1ac5164d15b87795eca94742f593f7b0078630ab4ae4ec50836e8c5c1a7" gracePeriod=30
Sep 08 21:01:06.644913 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:06.644886    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"
Sep 08 21:01:06.756144 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:06.756108    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-metrics-certs\") pod \"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c\" (UID: \"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c\") "
Sep 08 21:01:06.756144 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:06.756144    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-maas-api-tls\") pod \"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c\" (UID: \"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c\") "
Sep 08 21:01:06.756450 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:06.756209    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-455s4\" (UniqueName: \"kubernetes.io/projected/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-kube-api-access-455s4\") pod \"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c\" (UID: \"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c\") "
Sep 08 21:01:06.756450 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:06.756263    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-openshift-service-ca\") pod \"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c\" (UID: \"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c\") "
Sep 08 21:01:06.756658 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:06.756630    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a4bd910f-2aa2-45cc-9f56-d4d98392ac4c" (UID: "a4bd910f-2aa2-45cc-9f56-d4d98392ac4c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 21:01:06.758434 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:06.758409    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a4bd910f-2aa2-45cc-9f56-d4d98392ac4c" (UID: "a4bd910f-2aa2-45cc-9f56-d4d98392ac4c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 21:01:06.758541 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:06.758425    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a4bd910f-2aa2-45cc-9f56-d4d98392ac4c" (UID: "a4bd910f-2aa2-45cc-9f56-d4d98392ac4c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 21:01:06.758541 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:06.758432    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-kube-api-access-455s4" (OuterVolumeSpecName: "kube-api-access-455s4") pod "a4bd910f-2aa2-45cc-9f56-d4d98392ac4c" (UID: "a4bd910f-2aa2-45cc-9f56-d4d98392ac4c"). InnerVolumeSpecName "kube-api-access-455s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 21:01:06.857543 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:06.857512    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-metrics-certs\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 21:01:06.857543 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:06.857537    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-maas-api-tls\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 21:01:06.857543 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:06.857546    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-455s4\" (UniqueName: \"kubernetes.io/projected/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-kube-api-access-455s4\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 21:01:06.857754 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:06.857558    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c-openshift-service-ca\") on node \"ip-10-0-129-118.ec2.internal\" DevicePath \"\""
Sep 08 21:01:07.366122 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:07.366084    2552 generic.go:356] "Generic (PLEG): container finished" podID="a4bd910f-2aa2-45cc-9f56-d4d98392ac4c" containerID="4f63d1ac5164d15b87795eca94742f593f7b0078630ab4ae4ec50836e8c5c1a7" exitCode=0
Sep 08 21:01:07.366302 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:07.366136    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"
Sep 08 21:01:07.366302 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:07.366167    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms" event={"ID":"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c","Type":"ContainerDied","Data":"4f63d1ac5164d15b87795eca94742f593f7b0078630ab4ae4ec50836e8c5c1a7"}
Sep 08 21:01:07.366302 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:07.366222    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms" event={"ID":"a4bd910f-2aa2-45cc-9f56-d4d98392ac4c","Type":"ContainerDied","Data":"f971fefcf09ea888b98e26feef2e6714aeb37e195f4e5d3207cbcc2ccc92df0c"}
Sep 08 21:01:07.366302 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:07.366260    2552 scope.go:117] "RemoveContainer" containerID="4f63d1ac5164d15b87795eca94742f593f7b0078630ab4ae4ec50836e8c5c1a7"
Sep 08 21:01:07.374499 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:07.374482    2552 scope.go:117] "RemoveContainer" containerID="4f63d1ac5164d15b87795eca94742f593f7b0078630ab4ae4ec50836e8c5c1a7"
Sep 08 21:01:07.374786 ip-10-0-129-118 kubenswrapper[2552]: E0908 21:01:07.374768    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f63d1ac5164d15b87795eca94742f593f7b0078630ab4ae4ec50836e8c5c1a7\": container with ID starting with 4f63d1ac5164d15b87795eca94742f593f7b0078630ab4ae4ec50836e8c5c1a7 not found: ID does not exist" containerID="4f63d1ac5164d15b87795eca94742f593f7b0078630ab4ae4ec50836e8c5c1a7"
Sep 08 21:01:07.374834 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:07.374794    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f63d1ac5164d15b87795eca94742f593f7b0078630ab4ae4ec50836e8c5c1a7"} err="failed to get container status \"4f63d1ac5164d15b87795eca94742f593f7b0078630ab4ae4ec50836e8c5c1a7\": rpc error: code = NotFound desc = could not find container \"4f63d1ac5164d15b87795eca94742f593f7b0078630ab4ae4ec50836e8c5c1a7\": container with ID starting with 4f63d1ac5164d15b87795eca94742f593f7b0078630ab4ae4ec50836e8c5c1a7 not found: ID does not exist"
Sep 08 21:01:07.384573 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:07.384552    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"]
Sep 08 21:01:07.387445 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:07.387427    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b96cdc7c6-6hrms"]
Sep 08 21:01:09.224235 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:09.224200    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4bd910f-2aa2-45cc-9f56-d4d98392ac4c" path="/var/lib/kubelet/pods/a4bd910f-2aa2-45cc-9f56-d4d98392ac4c/volumes"
Sep 08 21:01:11.705024 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:11.704982    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 21:01:17.698415 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:17.698370    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2t78w"]
Sep 08 21:01:27.371013 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:27.370984    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbd9f8d48-6jpjl_2b591c3f-1022-494d-8b19-eceefee547d0/copy-manifests/0.log"
Sep 08 21:01:27.381358 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:27.381330    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbd9f8d48-6jpjl_2b591c3f-1022-494d-8b19-eceefee547d0/manager/0.log"
Sep 08 21:01:28.090675 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:28.090632    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6b8bc4d5d4-hvtll_3e5ea18d-12a7-4285-99d0-5a0fd3477b30/maas-api/0.log"
Sep 08 21:01:30.065605 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:30.065575    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2t78w_0149abc2-b2c6-4bb7-b05c-5a7aeed704a9/limitador/0.log"
Sep 08 21:01:30.693658 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:30.693626    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nbq9q_ac1d416b-0cc0-4214-9d2b-83d341f0faee/discovery/0.log"
Sep 08 21:01:31.873113 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:31.873080    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-fxvq7_bc17884e-e7e5-4760-8012-871ab8238e71/main/0.log"
Sep 08 21:01:32.226195 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:32.226123    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-wqn6s_21fdd723-6baa-40dd-af59-27f850ed2fc6/main/0.log"
Sep 08 21:01:32.352386 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:32.352357    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phww_68fd636d-af9b-4192-886f-bac53ce12753/main/0.log"
Sep 08 21:01:32.472580 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:32.472546    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-ktbjp_9c1d751c-6450-4c91-8329-3cce55243335/main/0.log"
Sep 08 21:01:41.473896 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:41.473796    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f5v7c_0dc52e01-2095-40dd-8603-0fdf9d6ff1aa/global-pull-secret-syncer/0.log"
Sep 08 21:01:41.685421 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:41.685391    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xfczt_6d616a62-5fd9-40e1-b415-a4cd644a83e8/konnectivity-agent/0.log"
Sep 08 21:01:41.708079 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:41.708056    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-118.ec2.internal_791cc5370b298e2f89744b04dce519c9/haproxy/0.log"
Sep 08 21:01:46.388039 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:46.388002    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2t78w_0149abc2-b2c6-4bb7-b05c-5a7aeed704a9/limitador/0.log"
Sep 08 21:01:47.464828 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:47.464799    2552 scope.go:117] "RemoveContainer" containerID="20cce19ca67f54191730ee757989d4f6421427042e5066f87cf5ce9b3d2fd8f2"
Sep 08 21:01:47.472881 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:47.472858    2552 scope.go:117] "RemoveContainer" containerID="80f6261199602b099536665f395893aeba9d51ee135741a1f952b28608fdeb94"
Sep 08 21:01:48.546395 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:48.546313    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w94bw_2131df78-3591-4c7f-8b0d-4e3653471b83/node-exporter/0.log"
Sep 08 21:01:48.578745 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:48.578716    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w94bw_2131df78-3591-4c7f-8b0d-4e3653471b83/kube-rbac-proxy/0.log"
Sep 08 21:01:48.642558 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:48.642533    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w94bw_2131df78-3591-4c7f-8b0d-4e3653471b83/init-textfile/0.log"
Sep 08 21:01:48.916925 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:48.916902    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_de56a525-9845-4b83-8c6f-488b1514803e/prometheus/0.log"
Sep 08 21:01:48.939615 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:48.939582    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_de56a525-9845-4b83-8c6f-488b1514803e/config-reloader/0.log"
Sep 08 21:01:48.963435 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:48.963415    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_de56a525-9845-4b83-8c6f-488b1514803e/thanos-sidecar/0.log"
Sep 08 21:01:48.985561 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:48.985541    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_de56a525-9845-4b83-8c6f-488b1514803e/kube-rbac-proxy-web/0.log"
Sep 08 21:01:49.009002 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:49.008974    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_de56a525-9845-4b83-8c6f-488b1514803e/kube-rbac-proxy/0.log"
Sep 08 21:01:49.033471 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:49.033451    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_de56a525-9845-4b83-8c6f-488b1514803e/kube-rbac-proxy-thanos/0.log"
Sep 08 21:01:49.059331 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:49.059309    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_de56a525-9845-4b83-8c6f-488b1514803e/init-config-reloader/0.log"
Sep 08 21:01:51.995782 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:51.995748    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b"]
Sep 08 21:01:51.996130 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:51.996072    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4bd910f-2aa2-45cc-9f56-d4d98392ac4c" containerName="maas-api"
Sep 08 21:01:51.996130 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:51.996084    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4bd910f-2aa2-45cc-9f56-d4d98392ac4c" containerName="maas-api"
Sep 08 21:01:51.996211 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:51.996157    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4bd910f-2aa2-45cc-9f56-d4d98392ac4c" containerName="maas-api"
Sep 08 21:01:52.000626 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:52.000610    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b"
Sep 08 21:01:52.003222 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:52.003200    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w75tg\"/\"kube-root-ca.crt\""
Sep 08 21:01:52.004164 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:52.004144    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w75tg\"/\"openshift-service-ca.crt\""
Sep 08 21:01:52.004279 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:52.004149    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w75tg\"/\"default-dockercfg-xmph6\""
Sep 08 21:01:52.009025 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:52.009000    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b"]
Sep 08 21:01:52.128720 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:52.128684    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0bce2e1-e3c3-454b-a34c-014ea11f5065-sys\") pod \"perf-node-gather-daemonset-mz28b\" (UID: \"a0bce2e1-e3c3-454b-a34c-014ea11f5065\") " pod="openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b"
Sep 08 21:01:52.128720 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:52.128718    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a0bce2e1-e3c3-454b-a34c-014ea11f5065-podres\") pod \"perf-node-gather-daemonset-mz28b\" (UID: \"a0bce2e1-e3c3-454b-a34c-014ea11f5065\") " pod="openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b"
Sep 08 21:01:52.128924 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:52.128785    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f46ls\" (UniqueName: \"kubernetes.io/projected/a0bce2e1-e3c3-454b-a34c-014ea11f5065-kube-api-access-f46ls\") pod \"perf-node-gather-daemonset-mz28b\" (UID: \"a0bce2e1-e3c3-454b-a34c-014ea11f5065\") " pod="openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b"
Sep 08 21:01:52.128924 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:52.128830    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0bce2e1-e3c3-454b-a34c-014ea11f5065-lib-modules\") pod \"perf-node-gather-daemonset-mz28b\" (UID: \"a0bce2e1-e3c3-454b-a34c-014ea11f5065\") " pod="openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b"
Sep 08 21:01:52.128924 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:52.128855    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a0bce2e1-e3c3-454b-a34c-014ea11f5065-proc\") pod \"perf-node-gather-daemonset-mz28b\" (UID: \"a0bce2e1-e3c3-454b-a34c-014ea11f5065\") " pod="openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b"
Sep 08 21:01:52.229368 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:52.229337    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f46ls\" (UniqueName: \"kubernetes.io/projected/a0bce2e1-e3c3-454b-a34c-014ea11f5065-kube-api-access-f46ls\") pod \"perf-node-gather-daemonset-mz28b\" (UID: \"a0bce2e1-e3c3-454b-a34c-014ea11f5065\") " pod="openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b"
Sep 08 21:01:52.229492 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:52.229378    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0bce2e1-e3c3-454b-a34c-014ea11f5065-lib-modules\") pod \"perf-node-gather-daemonset-mz28b\" (UID: \"a0bce2e1-e3c3-454b-a34c-014ea11f5065\") " pod="openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b"
Sep 08 21:01:52.229492 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:52.229406    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a0bce2e1-e3c3-454b-a34c-014ea11f5065-proc\") pod \"perf-node-gather-daemonset-mz28b\" (UID: \"a0bce2e1-e3c3-454b-a34c-014ea11f5065\") " pod="openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b"
Sep 08 21:01:52.229626 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:52.229491    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0bce2e1-e3c3-454b-a34c-014ea11f5065-sys\") pod \"perf-node-gather-daemonset-mz28b\" (UID: \"a0bce2e1-e3c3-454b-a34c-014ea11f5065\") " pod="openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b"
Sep 08 21:01:52.229626 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:52.229515    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a0bce2e1-e3c3-454b-a34c-014ea11f5065-podres\") pod \"perf-node-gather-daemonset-mz28b\" (UID: \"a0bce2e1-e3c3-454b-a34c-014ea11f5065\") " pod="openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b"
Sep 08 21:01:52.229626 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:52.229529    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0bce2e1-e3c3-454b-a34c-014ea11f5065-lib-modules\") pod \"perf-node-gather-daemonset-mz28b\" (UID: \"a0bce2e1-e3c3-454b-a34c-014ea11f5065\") " pod="openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b"
Sep 08 21:01:52.229626 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:52.229557    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a0bce2e1-e3c3-454b-a34c-014ea11f5065-proc\") pod \"perf-node-gather-daemonset-mz28b\" (UID: \"a0bce2e1-e3c3-454b-a34c-014ea11f5065\") " pod="openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b"
Sep 08 21:01:52.229626 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:52.229592    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0bce2e1-e3c3-454b-a34c-014ea11f5065-sys\") pod \"perf-node-gather-daemonset-mz28b\" (UID: \"a0bce2e1-e3c3-454b-a34c-014ea11f5065\") " pod="openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b"
Sep 08 21:01:52.229626 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:52.229610    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a0bce2e1-e3c3-454b-a34c-014ea11f5065-podres\") pod \"perf-node-gather-daemonset-mz28b\" (UID: \"a0bce2e1-e3c3-454b-a34c-014ea11f5065\") " pod="openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b"
Sep 08 21:01:52.237813 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:52.237792    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f46ls\" (UniqueName: \"kubernetes.io/projected/a0bce2e1-e3c3-454b-a34c-014ea11f5065-kube-api-access-f46ls\") pod \"perf-node-gather-daemonset-mz28b\" (UID: \"a0bce2e1-e3c3-454b-a34c-014ea11f5065\") " pod="openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b"
Sep 08 21:01:52.310995 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:52.310928    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b"
Sep 08 21:01:52.580164 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:52.580137    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b"]
Sep 08 21:01:52.582461 ip-10-0-129-118 kubenswrapper[2552]: W0908 21:01:52.582424    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda0bce2e1_e3c3_454b_a34c_014ea11f5065.slice/crio-fb3d1a654fec348dc27cc29c6fdc18ba7db92985642e887f3cdfcf5a400c48f4 WatchSource:0}: Error finding container fb3d1a654fec348dc27cc29c6fdc18ba7db92985642e887f3cdfcf5a400c48f4: Status 404 returned error can't find the container with id fb3d1a654fec348dc27cc29c6fdc18ba7db92985642e887f3cdfcf5a400c48f4
Sep 08 21:01:53.563541 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:53.563509    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b" event={"ID":"a0bce2e1-e3c3-454b-a34c-014ea11f5065","Type":"ContainerStarted","Data":"2adc37e4d980a11627cfbab8dd81f61d453bd891a7282d75712ed49ca76c143d"}
Sep 08 21:01:53.563909 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:53.563565    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b" event={"ID":"a0bce2e1-e3c3-454b-a34c-014ea11f5065","Type":"ContainerStarted","Data":"fb3d1a654fec348dc27cc29c6fdc18ba7db92985642e887f3cdfcf5a400c48f4"}
Sep 08 21:01:53.563909 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:53.563598    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b"
Sep 08 21:01:53.585332 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:53.585283    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b" podStartSLOduration=2.585268426 podStartE2EDuration="2.585268426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 21:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 21:01:53.583944251 +0000 UTC m=+2166.835425608" watchObservedRunningTime="2026-09-08 21:01:53.585268426 +0000 UTC m=+2166.836749827"
Sep 08 21:01:54.124750 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:54.124716    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pfxtz_044d382c-a15c-462b-8cd1-6c7396d46f3a/dns/0.log"
Sep 08 21:01:54.143632 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:54.143605    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pfxtz_044d382c-a15c-462b-8cd1-6c7396d46f3a/kube-rbac-proxy/0.log"
Sep 08 21:01:54.251067 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:54.251044    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-th7xh_bced89cc-7b97-493f-b71d-8fc32bc7187f/dns-node-resolver/0.log"
Sep 08 21:01:54.852070 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:54.852038    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zxmp4_345de38f-94c6-4067-a5c6-fc7ad6a560e9/node-ca/0.log"
Sep 08 21:01:56.195058 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:56.195020    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nbq9q_ac1d416b-0cc0-4214-9d2b-83d341f0faee/discovery/0.log"
Sep 08 21:01:57.310096 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:57.310065    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vdrmg_d8c650d9-07af-47a5-80d7-dd191f995c7f/serve-healthcheck-canary/0.log"
Sep 08 21:01:57.935265 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:57.935221    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kw8zt_607dd30a-5e39-4cc8-91b0-cba0ad740873/kube-rbac-proxy/0.log"
Sep 08 21:01:57.954441 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:57.954419    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kw8zt_607dd30a-5e39-4cc8-91b0-cba0ad740873/exporter/0.log"
Sep 08 21:01:57.974701 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:57.974675    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kw8zt_607dd30a-5e39-4cc8-91b0-cba0ad740873/extractor/0.log"
Sep 08 21:01:59.578035 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:01:59.578006    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-w75tg/perf-node-gather-daemonset-mz28b"
Sep 08 21:02:00.532085 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:02:00.532055    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbd9f8d48-6jpjl_2b591c3f-1022-494d-8b19-eceefee547d0/manager/0.log"
Sep 08 21:02:00.553086 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:02:00.553066    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbd9f8d48-6jpjl_2b591c3f-1022-494d-8b19-eceefee547d0/copy-manifests/0.log"
Sep 08 21:02:02.257752 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:02:02.257719    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-57c49bb89d-qmlxh_0bf6a7f6-3bf9-4000-9eec-099d382568b8/manager/0.log"
Sep 08 21:02:03.183595 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:02:03.183558    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nbq9q_ac1d416b-0cc0-4214-9d2b-83d341f0faee/discovery/0.log"
Sep 08 21:02:05.193623 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:02:05.193584    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nbq9q_ac1d416b-0cc0-4214-9d2b-83d341f0faee/discovery/0.log"
Sep 08 21:02:07.336719 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:02:07.336690    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-lghl7_3fa62b2d-92ab-4958-83a4-5fa1c56ea4cd/migrator/0.log"
Sep 08 21:02:07.357787 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:02:07.357756    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-lghl7_3fa62b2d-92ab-4958-83a4-5fa1c56ea4cd/graceful-termination/0.log"
Sep 08 21:02:08.790493 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:02:08.790461    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9fmtz_0fd7b065-bbbb-458a-840d-da97ea47930a/kube-multus/0.log"
Sep 08 21:02:09.132401 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:02:09.132374    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvlkf_135336c8-5355-4a06-bd24-30cf54fd420a/kube-multus-additional-cni-plugins/0.log"
Sep 08 21:02:09.154875 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:02:09.154843    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvlkf_135336c8-5355-4a06-bd24-30cf54fd420a/egress-router-binary-copy/0.log"
Sep 08 21:02:09.176520 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:02:09.176501    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvlkf_135336c8-5355-4a06-bd24-30cf54fd420a/cni-plugins/0.log"
Sep 08 21:02:09.196942 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:02:09.196919    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvlkf_135336c8-5355-4a06-bd24-30cf54fd420a/bond-cni-plugin/0.log"
Sep 08 21:02:09.216381 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:02:09.216359    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvlkf_135336c8-5355-4a06-bd24-30cf54fd420a/routeoverride-cni/0.log"
Sep 08 21:02:09.237245 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:02:09.237221    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvlkf_135336c8-5355-4a06-bd24-30cf54fd420a/whereabouts-cni-bincopy/0.log"
Sep 08 21:02:09.260337 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:02:09.260315    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvlkf_135336c8-5355-4a06-bd24-30cf54fd420a/whereabouts-cni/0.log"
Sep 08 21:02:09.478392 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:02:09.478309    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zwvng_0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059/network-metrics-daemon/0.log"
Sep 08 21:02:09.494578 ip-10-0-129-118 kubenswrapper[2552]: I0908 21:02:09.494552    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zwvng_0f58e4f0-1c4d-4ac4-8e36-f3b1cf7e5059/kube-rbac-proxy/0.log"