Sep 09 20:30:27.213965 ip-10-0-135-117 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 20:30:27.213979 ip-10-0-135-117 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 20:30:27.213989 ip-10-0-135-117 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 20:30:27.214200 ip-10-0-135-117 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 20:30:37.462907 ip-10-0-135-117 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 20:30:37.462923 ip-10-0-135-117 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b3ed82dfe9ce4426a8aa1d3844d8d449 --
Sep 09 20:32:32.892892 ip-10-0-135-117 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 20:32:33.229945 ip-10-0-135-117 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:32:33.229945 ip-10-0-135-117 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 20:32:33.229945 ip-10-0-135-117 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:32:33.229945 ip-10-0-135-117 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 20:32:33.229945 ip-10-0-135-117 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:32:33.230767 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.230687    2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 20:32:33.234224 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234206    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:32:33.234224 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234221    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:32:33.234224 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234226    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:32:33.234224 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234230    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:32:33.234438 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234234    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:32:33.234438 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234238    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:32:33.234438 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234242    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:32:33.234438 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234247    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:32:33.234438 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234250    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:32:33.234438 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234254    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:32:33.234438 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234259    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:32:33.234438 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234262    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:32:33.234438 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234266    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:32:33.234438 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234270    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:32:33.234438 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234274    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:32:33.234438 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234278    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:32:33.234438 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234282    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:32:33.234438 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234286    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:32:33.234438 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234293    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:32:33.234438 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234299    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:32:33.234438 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234303    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:32:33.234438 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234308    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:32:33.234438 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234313    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:32:33.235234 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234317    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:32:33.235234 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234321    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:32:33.235234 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234325    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:32:33.235234 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234330    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:32:33.235234 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234335    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:32:33.235234 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234339    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:32:33.235234 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234350    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:32:33.235234 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234355    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:32:33.235234 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234359    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:32:33.235234 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234363    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:32:33.235234 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234368    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:32:33.235234 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234371    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:32:33.235234 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234376    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:32:33.235234 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234380    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:32:33.235234 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234385    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:32:33.235234 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234390    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:32:33.235234 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234394    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:32:33.235234 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234398    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:32:33.235234 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234403    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:32:33.235234 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234407    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:32:33.236145 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234411    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:32:33.236145 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234416    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:32:33.236145 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234420    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:32:33.236145 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234424    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:32:33.236145 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234428    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:32:33.236145 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234433    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:32:33.236145 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234436    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:32:33.236145 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234441    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:32:33.236145 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234445    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:32:33.236145 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234450    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:32:33.236145 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234454    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:32:33.236145 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234458    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:32:33.236145 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234462    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:32:33.236145 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234466    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:32:33.236145 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234470    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:32:33.236145 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234474    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:32:33.236145 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234479    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:32:33.236145 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234484    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:32:33.236145 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234488    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:32:33.236145 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234492    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:32:33.237008 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234498    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:32:33.237008 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234502    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:32:33.237008 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234507    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:32:33.237008 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234511    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:32:33.237008 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234515    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:32:33.237008 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234522    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:32:33.237008 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234528    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:32:33.237008 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234533    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:32:33.237008 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234538    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:32:33.237008 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234543    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:32:33.237008 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234548    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:32:33.237008 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234552    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:32:33.237008 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234557    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:32:33.237008 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234561    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:32:33.237008 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234566    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:32:33.237008 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234570    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:32:33.237008 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234575    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:32:33.237008 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234580    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:32:33.237008 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234598    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:32:33.237832 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234603    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:32:33.237832 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234607    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:32:33.237832 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234611    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:32:33.237832 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234615    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:32:33.237832 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234620    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:32:33.237832 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234624    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:32:33.237832 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234628    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:32:33.237832 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234637    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:32:33.237832 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234642    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:32:33.237832 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234647    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:32:33.237832 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234651    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:32:33.237832 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234655    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:32:33.237832 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234659    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:32:33.237832 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234663    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:32:33.237832 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234667    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:32:33.237832 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234672    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:32:33.237832 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.234676    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:32:33.237832 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235795    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:32:33.237832 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235805    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:32:33.237832 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235812    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:32:33.238716 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235816    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:32:33.238716 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235821    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:32:33.238716 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235825    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:32:33.238716 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235829    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:32:33.238716 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235834    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:32:33.238716 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235838    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:32:33.238716 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235842    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:32:33.238716 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235846    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:32:33.238716 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235850    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:32:33.238716 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235856    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:32:33.238716 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235862    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:32:33.238716 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235866    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:32:33.238716 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235870    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:32:33.238716 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235875    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:32:33.238716 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235879    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:32:33.238716 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235884    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:32:33.238716 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235888    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:32:33.238716 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235892    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:32:33.238716 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235896    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:32:33.238716 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235900    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:32:33.239353 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235904    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:32:33.239353 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235908    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:32:33.239353 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235913    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:32:33.239353 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235917    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:32:33.239353 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235921    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:32:33.239353 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235925    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:32:33.239353 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235929    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:32:33.239353 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235933    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:32:33.239353 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235937    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:32:33.239353 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235941    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:32:33.239353 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235945    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:32:33.239353 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235953    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:32:33.239353 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235960    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:32:33.239353 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235964    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:32:33.239353 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235969    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:32:33.239353 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235974    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:32:33.239353 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235978    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:32:33.239353 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235985    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:32:33.239353 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235990    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:32:33.239879 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.235995    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:32:33.239879 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236000    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:32:33.239879 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236004    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:32:33.239879 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236008    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:32:33.239879 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236013    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:32:33.239879 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236018    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:32:33.239879 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236022    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:32:33.239879 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236026    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:32:33.239879 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236030    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:32:33.239879 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236034    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:32:33.239879 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236039    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:32:33.239879 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236043    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:32:33.239879 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236047    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:32:33.239879 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236051    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:32:33.239879 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236055    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:32:33.239879 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236059    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:32:33.239879 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236063    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:32:33.239879 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236067    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:32:33.239879 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236071    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:32:33.240501 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236078    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:32:33.240501 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236084    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:32:33.240501 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236089    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:32:33.240501 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236094    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:32:33.240501 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236098    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:32:33.240501 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236103    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:32:33.240501 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236108    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:32:33.240501 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236112    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:32:33.240501 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236116    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:32:33.240501 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236121    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:32:33.240501 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236125    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:32:33.240501 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236129    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:32:33.240501 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236133    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:32:33.240501 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236137    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:32:33.240501 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236141    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:32:33.240501 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236145    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:32:33.240501 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236149    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:32:33.240501 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236153    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:32:33.240501 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236158    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:32:33.241061 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236162    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:32:33.241061 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236168    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:32:33.241061 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236172    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:32:33.241061 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236177    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:32:33.241061 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236181    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:32:33.241061 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236185    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:32:33.241061 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236189    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:32:33.241061 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236193    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:32:33.241061 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236197    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:32:33.241061 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236201    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:32:33.241061 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236206    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:32:33.241061 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236211    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:32:33.241061 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236215    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:32:33.241061 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236219    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:32:33.241061 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236223    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:32:33.241061 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236228    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:32:33.241061 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236232    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:32:33.241061 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236236    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:32:33.241061 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236241    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:32:33.241061 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236833    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:32:33.241607 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236842    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:32:33.241607 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236847    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:32:33.241607 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236851    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:32:33.241607 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236856    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:32:33.241607 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236863    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:32:33.241607 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236869    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:32:33.241607 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236874    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:32:33.241607 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236879    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:32:33.241607 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236884    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:32:33.241607 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236888    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:32:33.241607 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236893    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:32:33.241607 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236898    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:32:33.241607 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236902    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:32:33.241607 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236907    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:32:33.241607 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236911    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:32:33.241607 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236915    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:32:33.241607 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236919    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:32:33.241607 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236923    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:32:33.241607 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236927    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:32:33.242200 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236931    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:32:33.242200 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236936    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:32:33.242200 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236940    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:32:33.242200 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236945    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:32:33.242200 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236950    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:32:33.242200 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236954    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:32:33.242200 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236958    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:32:33.242200 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236963    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:32:33.242200 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236967    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:32:33.242200 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236971    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:32:33.242200 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236976    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:32:33.242200 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236983    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:32:33.242200 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236988    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:32:33.242200 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236992    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:32:33.242200 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.236996    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:32:33.242200 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237001    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:32:33.242200 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237004    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:32:33.242200 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237008    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:32:33.242200 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237012    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:32:33.242200 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237016    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:32:33.242727 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237020    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:32:33.242727 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237025    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:32:33.242727 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237028    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:32:33.242727 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237033    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:32:33.242727 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237037    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:32:33.242727 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237044    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:32:33.242727 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237049    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:32:33.242727 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237053    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:32:33.242727 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237057    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:32:33.242727 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237062    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:32:33.242727 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237066    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:32:33.242727 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237070    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:32:33.242727 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237074    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:32:33.242727 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237078    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:32:33.242727 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237082    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:32:33.242727 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237086    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:32:33.242727 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237090    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:32:33.242727 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237095    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:32:33.242727 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237099    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:32:33.242727 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237103    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:32:33.243210 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237107    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:32:33.243210 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237112    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:32:33.243210 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237116    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:32:33.243210 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237123    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:32:33.243210 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237128    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:32:33.243210 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237132    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:32:33.243210 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237136    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:32:33.243210 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237143    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:32:33.243210 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237149    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:32:33.243210 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237154    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:32:33.243210 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237158    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:32:33.243210 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237162    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:32:33.243210 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237167    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:32:33.243210 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237171    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:32:33.243210 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237175    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:32:33.243210 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237179    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:32:33.243210 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237184    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:32:33.243210 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237188    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:32:33.243210 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237193    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:32:33.243720 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237197    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:32:33.243720 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237202    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:32:33.243720 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237206    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:32:33.243720 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237210    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:32:33.243720 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237215    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:32:33.243720 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237219    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:32:33.243720 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237223    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:32:33.243720 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237227    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:32:33.243720 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237231    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:32:33.243720 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237235    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:32:33.243720 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237239    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:32:33.243720 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237243    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:32:33.243720 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237248    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:32:33.243720 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237251    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:32:33.243720 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237255    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:32:33.243720 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237262    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:32:33.243720 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237271    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:32:33.243720 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237276    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:32:33.243720 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237281    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.237286    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237391    2556 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237438    2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237448    2556 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237455    2556 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237461    2556 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237466    2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237473    2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237480    2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237485    2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237489    2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237495    2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237500    2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237506    2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237511    2556 flags.go:64] FLAG: --cgroup-root=""
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237516    2556 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237520    2556 flags.go:64] FLAG: --client-ca-file=""
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237525    2556 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237530    2556 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237537    2556 flags.go:64] FLAG: --cluster-domain=""
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237542    2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237547    2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237552    2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 20:32:33.244181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237557    2556 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237563    2556 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237568    2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237572    2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237578    2556 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237600    2556 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237609    2556 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237614    2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237618    2556 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237623    2556 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237633    2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237638    2556 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237642    2556 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237647    2556 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237652    2556 flags.go:64] FLAG: --enable-server="true"
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237656    2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237662    2556 flags.go:64] FLAG: --event-burst="100"
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237667    2556 flags.go:64] FLAG: --event-qps="50"
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237672    2556 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237677    2556 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237681    2556 flags.go:64] FLAG: --eviction-hard=""
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237687    2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237692    2556 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237698    2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237703    2556 flags.go:64] FLAG: --eviction-soft=""
Sep 09 20:32:33.244840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237707    2556 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 20:32:33.245443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237712    2556 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 20:32:33.245443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237716    2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 20:32:33.245443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237721    2556 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 20:32:33.245443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237725    2556 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 20:32:33.245443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237730    2556 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 20:32:33.245443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237735    2556 flags.go:64] FLAG: --feature-gates=""
Sep 09 20:32:33.245443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237740    2556 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 20:32:33.245443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237745    2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 20:32:33.245443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237750    2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 20:32:33.245443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237755    2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 20:32:33.245443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237760    2556 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 20:32:33.245443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237765    2556 flags.go:64] FLAG: --help="false"
Sep 09 20:32:33.245443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237773    2556 flags.go:64] FLAG: --hostname-override="ip-10-0-135-117.ec2.internal"
Sep 09 20:32:33.245443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237778    2556 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 20:32:33.245443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237783    2556 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 20:32:33.245443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237787    2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 20:32:33.245443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237793    2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 20:32:33.245443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237798    2556 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 20:32:33.245443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237803    2556 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 20:32:33.245443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237808    2556 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 20:32:33.245443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237812    2556 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 20:32:33.245443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237817    2556 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 20:32:33.245443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237821    2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237827    2556 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237831    2556 flags.go:64] FLAG: --kube-reserved=""
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237835    2556 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237840    2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237844    2556 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237848    2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237853    2556 flags.go:64] FLAG: --lock-file=""
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237861    2556 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237866    2556 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237871    2556 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237880    2556 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237885    2556 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237889    2556 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237894    2556 flags.go:64] FLAG: --logging-format="text"
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237898    2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237903    2556 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237908    2556 flags.go:64] FLAG: --manifest-url=""
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237913    2556 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237919    2556 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237924    2556 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237930    2556 flags.go:64] FLAG: --max-pods="110"
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237937    2556 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237942    2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237946    2556 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 20:32:33.246072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237951    2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237956    2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237960    2556 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237965    2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237977    2556 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237981    2556 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237986    2556 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237991    2556 flags.go:64] FLAG: --pod-cidr=""
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.237996    2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238004    2556 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238009    2556 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238014    2556 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238019    2556 flags.go:64] FLAG: --port="10250"
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238023    2556 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238028    2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dc0ce8dd846d5c8c"
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238033    2556 flags.go:64] FLAG: --qos-reserved=""
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238039    2556 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238044    2556 flags.go:64] FLAG: --register-node="true"
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238048    2556 flags.go:64] FLAG: --register-with-taints=""
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238054    2556 flags.go:64] FLAG: --registry-burst="10"
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238058    2556 flags.go:64] FLAG: --registry-qps="5"
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238063    2556 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238067    2556 flags.go:64] FLAG: --reserved-memory=""
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238074    2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238079    2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 20:32:33.246700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238084    2556 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238088    2556 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238093    2556 flags.go:64] FLAG: --runonce="false"
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238097    2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238104    2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238109    2556 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238114    2556 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238119    2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238124    2556 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238129    2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238133    2556 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238138    2556 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238142    2556 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238147    2556 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238152    2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238157    2556 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238161    2556 flags.go:64] FLAG: --system-cgroups=""
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238166    2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238175    2556 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238179    2556 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238191    2556 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238198    2556 flags.go:64] FLAG: --tls-min-version=""
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238203    2556 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238208    2556 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238213    2556 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 20:32:33.247310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238218    2556 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 20:32:33.247957 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238223    2556 flags.go:64] FLAG: --v="2"
Sep 09 20:32:33.247957 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238229    2556 flags.go:64] FLAG: --version="false"
Sep 09 20:32:33.247957 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238235    2556 flags.go:64] FLAG: --vmodule=""
Sep 09 20:32:33.247957 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238241    2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 20:32:33.247957 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238246    2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 20:32:33.247957 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238418    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:32:33.247957 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238424    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:32:33.247957 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238429    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:32:33.247957 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238433    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:32:33.247957 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238439    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:32:33.247957 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238448    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:32:33.247957 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238453    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:32:33.247957 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238457    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:32:33.247957 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238461    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:32:33.247957 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238465    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:32:33.247957 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238469    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:32:33.247957 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238474    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:32:33.247957 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238478    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:32:33.247957 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238482    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:32:33.247957 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238487    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:32:33.247957 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238491    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:32:33.248464 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238495    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:32:33.248464 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238499    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:32:33.248464 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238506    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:32:33.248464 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238512    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:32:33.248464 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238517    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:32:33.248464 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238521    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:32:33.248464 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238526    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:32:33.248464 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238532    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:32:33.248464 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238537    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:32:33.248464 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238542    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:32:33.248464 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238547    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:32:33.248464 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238551    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:32:33.248464 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238555    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:32:33.248464 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238560    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:32:33.248464 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238564    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:32:33.248464 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238568    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:32:33.248464 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238572    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:32:33.248464 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238576    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:32:33.248464 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238580    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:32:33.248945 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238599    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:32:33.248945 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238604    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:32:33.248945 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238610    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:32:33.248945 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238615    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:32:33.248945 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238619    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:32:33.248945 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238624    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:32:33.248945 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238628    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:32:33.248945 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238632    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:32:33.248945 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238636    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:32:33.248945 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238640    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:32:33.248945 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238645    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:32:33.248945 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238649    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:32:33.248945 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238653    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:32:33.248945 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238657    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:32:33.248945 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238661    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:32:33.248945 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238665    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:32:33.248945 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238669    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:32:33.248945 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238674    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:32:33.248945 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238678    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:32:33.249401 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238683    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:32:33.249401 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238687    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:32:33.249401 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238692    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:32:33.249401 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238696    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:32:33.249401 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238700    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:32:33.249401 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238705    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:32:33.249401 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238709    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:32:33.249401 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238713    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:32:33.249401 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238717    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:32:33.249401 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238721    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:32:33.249401 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238725    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:32:33.249401 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238729    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:32:33.249401 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238733    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:32:33.249401 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238737    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:32:33.249401 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238741    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:32:33.249401 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238747    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:32:33.249401 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238753    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:32:33.249401 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238758    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:32:33.249401 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238762    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:32:33.249401 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238766    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:32:33.249944 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238770    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:32:33.249944 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238777    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:32:33.249944 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238783    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:32:33.249944 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238788    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:32:33.249944 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238793    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:32:33.249944 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238797    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:32:33.249944 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238801    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:32:33.249944 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238806    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:32:33.249944 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238810    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:32:33.249944 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238814    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:32:33.249944 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238819    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:32:33.249944 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238823    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:32:33.249944 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238827    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:32:33.249944 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238831    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:32:33.249944 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238836    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:32:33.249944 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238839    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:32:33.249944 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238843    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:32:33.249944 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238847    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:32:33.249944 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238852    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:32:33.249944 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238857    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:32:33.250437 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238861    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:32:33.250437 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238865    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:32:33.250437 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238869    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:32:33.250437 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238873    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:32:33.250437 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.238877    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:32:33.250437 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.238886    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:32:33.250437 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.245801    2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 20:32:33.250437 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.245816    2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 20:32:33.250437 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.246517    2556 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 20:32:33.250437 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.246526    2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 20:32:33.250437 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246581    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:32:33.250437 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246602    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:32:33.250437 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246606    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:32:33.250437 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246609    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:32:33.250819 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246613    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:32:33.250819 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246616    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:32:33.250819 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246618    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:32:33.250819 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246622    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:32:33.250819 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246624    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:32:33.250819 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246627    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:32:33.250819 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246631    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:32:33.250819 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246635    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:32:33.250819 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246638    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:32:33.250819 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246641    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:32:33.250819 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246644    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:32:33.250819 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246646    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:32:33.250819 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246649    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:32:33.250819 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246652    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:32:33.250819 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246655    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:32:33.250819 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246658    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:32:33.250819 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246660    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:32:33.250819 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246663    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:32:33.250819 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246666    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:32:33.250819 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246669    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:32:33.251303 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246671    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:32:33.251303 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246674    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:32:33.251303 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246677    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:32:33.251303 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246680    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:32:33.251303 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246683    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:32:33.251303 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246687    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:32:33.251303 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246690    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:32:33.251303 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246692    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:32:33.251303 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246695    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:32:33.251303 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246698    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:32:33.251303 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246701    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:32:33.251303 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246704    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:32:33.251303 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246706    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:32:33.251303 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246709    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:32:33.251303 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246712    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:32:33.251303 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246715    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:32:33.251303 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246717    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:32:33.251303 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246721    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:32:33.251303 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246723    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:32:33.251775 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246726    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:32:33.251775 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246729    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:32:33.251775 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246733    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:32:33.251775 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246737    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:32:33.251775 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246740    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:32:33.251775 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246744    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:32:33.251775 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246746    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:32:33.251775 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246749    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:32:33.251775 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246751    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:32:33.251775 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246754    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:32:33.251775 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246756    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:32:33.251775 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246759    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:32:33.251775 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246761    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:32:33.251775 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246764    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:32:33.251775 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246767    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:32:33.251775 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246769    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:32:33.251775 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246772    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:32:33.251775 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246775    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:32:33.251775 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246778    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:32:33.252249 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246781    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:32:33.252249 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246784    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:32:33.252249 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246786    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:32:33.252249 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246789    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:32:33.252249 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246791    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:32:33.252249 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246794    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:32:33.252249 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246797    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:32:33.252249 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246799    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:32:33.252249 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246802    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:32:33.252249 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246805    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:32:33.252249 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246807    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:32:33.252249 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246810    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:32:33.252249 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246813    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:32:33.252249 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246815    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:32:33.252249 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246818    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:32:33.252249 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246820    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:32:33.252249 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246823    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:32:33.252249 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246826    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:32:33.252249 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246828    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:32:33.252249 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246831    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:32:33.252760 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246833    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:32:33.252760 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246836    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:32:33.252760 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246839    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:32:33.252760 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246841    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:32:33.252760 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246844    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:32:33.252760 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246847    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:32:33.252760 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246851    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:32:33.252760 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246854    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:32:33.252760 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246857    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:32:33.252760 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246860    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:32:33.252760 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246863    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:32:33.252760 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246866    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:32:33.252760 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246869    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:32:33.252760 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246872    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:32:33.252760 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246874    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:32:33.252760 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246877    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:32:33.252760 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246880    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:32:33.253207 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.246885    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:32:33.253207 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246994    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:32:33.253207 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.246999    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:32:33.253207 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247002    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:32:33.253207 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247005    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:32:33.253207 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247007    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:32:33.253207 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247010    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:32:33.253207 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247013    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:32:33.253207 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247016    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:32:33.253207 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247019    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:32:33.253207 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247021    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:32:33.253207 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247024    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:32:33.253207 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247027    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:32:33.253207 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247029    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:32:33.253207 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247032    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:32:33.253661 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247034    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:32:33.253661 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247037    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:32:33.253661 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247040    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:32:33.253661 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247042    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:32:33.253661 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247045    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:32:33.253661 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247047    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:32:33.253661 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247050    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:32:33.253661 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247052    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:32:33.253661 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247055    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:32:33.253661 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247058    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:32:33.253661 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247061    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:32:33.253661 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247064    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:32:33.253661 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247067    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:32:33.253661 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247069    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:32:33.253661 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247072    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:32:33.253661 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247075    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:32:33.253661 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247077    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:32:33.253661 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247080    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:32:33.253661 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247082    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:32:33.253661 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247085    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:32:33.254365 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247087    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:32:33.254365 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247090    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:32:33.254365 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247092    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:32:33.254365 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247095    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:32:33.254365 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247097    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:32:33.254365 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247100    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:32:33.254365 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247102    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:32:33.254365 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247105    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:32:33.254365 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247108    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:32:33.254365 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247110    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:32:33.254365 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247113    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:32:33.254365 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247116    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:32:33.254365 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247118    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:32:33.254365 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247121    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:32:33.254365 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247123    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:32:33.254365 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247126    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:32:33.254365 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247128    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:32:33.254365 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247132    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:32:33.254365 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247136    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:32:33.255115 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247139    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:32:33.255115 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247142    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:32:33.255115 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247145    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:32:33.255115 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247148    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:32:33.255115 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247151    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:32:33.255115 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247154    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:32:33.255115 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247156    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:32:33.255115 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247159    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:32:33.255115 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247162    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:32:33.255115 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247165    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:32:33.255115 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247168    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:32:33.255115 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247172    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:32:33.255115 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247175    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:32:33.255115 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247178    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:32:33.255115 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247180    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:32:33.255115 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247183    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:32:33.255115 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247185    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:32:33.255115 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247188    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:32:33.255115 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247190    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:32:33.255115 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247193    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:32:33.255625 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247195    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:32:33.255625 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247198    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:32:33.255625 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247200    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:32:33.255625 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247203    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:32:33.255625 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247206    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:32:33.255625 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247208    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:32:33.255625 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247211    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:32:33.255625 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247213    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:32:33.255625 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247216    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:32:33.255625 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247219    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:32:33.255625 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247223    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:32:33.255625 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247226    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:32:33.255625 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247229    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:32:33.255625 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247231    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:32:33.255625 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247234    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:32:33.255625 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247237    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:32:33.255625 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247243    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:32:33.255625 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247246    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:32:33.255625 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247248    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:32:33.255625 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247251    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:32:33.256107 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247254    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:32:33.256107 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247257    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:32:33.256107 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247260    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:32:33.256107 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247263    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:32:33.256107 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247265    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:32:33.256107 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:33.247268    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:32:33.256107 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.247273    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:32:33.256107 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.247369    2556 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 20:32:33.256107 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.250213    2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 20:32:33.256107 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.250915    2556 server.go:1019] "Starting client certificate rotation"
Sep 09 20:32:33.256107 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.251016    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:32:33.256107 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.251051    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:32:33.256107 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.254201    2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:32:33.258759 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.258742    2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:32:33.268494 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.268477    2556 log.go:25] "Validated CRI v1 runtime API"
Sep 09 20:32:33.272663 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.272649    2556 log.go:25] "Validated CRI v1 image API"
Sep 09 20:32:33.273497 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.273481    2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 20:32:33.275634 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.275617    2556 fs.go:135] Filesystem UUIDs: map[4afaafd4-f26e-4c6f-8ec7-e6c7f88358d2:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b0345a9f-e678-496d-a886-4b09b00b8118:/dev/nvme0n1p4]
Sep 09 20:32:33.275682 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.275633    2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 20:32:33.280448 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.280347    2556 manager.go:216] Machine: {Timestamp:2026-09-09 20:32:33.279148735 +0000 UTC m=+0.285427748 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100275 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2943aa1748f833f5c3076e4410c5aa SystemUUID:ec2943aa-1748-f833-f5c3-076e4410c5aa BootID:b3ed82df-e9ce-4426-a8aa-1d3844d8d449 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:63:fa:c3:89:17 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:63:fa:c3:89:17 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:60:9d:ac:f7:03 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 20:32:33.280448 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.280446    2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 20:32:33.280553 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.280509    2556 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 20:32:33.283351 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.283327    2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 20:32:33.283473 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.283353    2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-117.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 20:32:33.283895 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.283885    2556 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 20:32:33.283931 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.283897    2556 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 20:32:33.283931 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.283905    2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:32:33.284350 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.284340    2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:32:33.285085 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.285074    2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 20:32:33.287581 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.287564    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:32:33.288700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.288687    2556 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:32:33.288814 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.288805    2556 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 20:32:33.288853 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.288832    2556 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 20:32:33.288853 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.288841    2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 20:32:33.288904 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.288858    2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 20:32:33.288904 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.288867    2556 kubelet.go:398] "Adding apiserver pod source"
Sep 09 20:32:33.288904 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.288875    2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 20:32:33.289222 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.289210    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:32:33.289260 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.289234    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:32:33.291192 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.291168    2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 20:32:33.293574 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.293562    2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 20:32:33.293635 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.293602    2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 20:32:33.294689 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.294678    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 20:32:33.294730 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.294693    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 20:32:33.294730 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.294698    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 20:32:33.294730 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.294703    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 20:32:33.294730 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.294711    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 20:32:33.294730 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.294716    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 20:32:33.294730 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.294729    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 20:32:33.294881 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.294735    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 20:32:33.294881 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.294741    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 20:32:33.294881 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.294746    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 20:32:33.294881 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.294760    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 20:32:33.294881 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.294765    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 20:32:33.295385 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.295375    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 20:32:33.295385 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.295385    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 20:32:33.298803 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.298783    2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-117.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 20:32:33.298882 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.298824    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-117.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 20:32:33.298882 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.298824    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 20:32:33.300444 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.300429    2556 server.go:1268] "Started kubelet"
Sep 09 20:32:33.300509 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.300490    2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 20:32:33.300558 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.300509    2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 20:32:33.300627 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.300572    2556 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 20:32:33.301308 ip-10-0-135-117 systemd[1]: Started Kubernetes Kubelet.
Sep 09 20:32:33.302329 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.301958    2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 20:32:33.303902 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.303885    2556 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 20:32:33.304335 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.304316    2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 20:32:33.306099 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.305377    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-117.ec2.internal.18d3c13efea234e2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-117.ec2.internal,UID:ip-10-0-135-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-117.ec2.internal,},FirstTimestamp:2026-09-09 20:32:33.300395234 +0000 UTC m=+0.306674247,LastTimestamp:2026-09-09 20:32:33.300395234 +0000 UTC m=+0.306674247,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-117.ec2.internal,}"
Sep 09 20:32:33.310298 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.310284    2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 20:32:33.310388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.310301    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 20:32:33.310388 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.310359    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-117.ec2.internal\" not found"
Sep 09 20:32:33.310806 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.310776    2556 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 20:32:33.310806 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.310796    2556 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 20:32:33.310940 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.310834    2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 20:32:33.310940 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.310873    2556 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 20:32:33.310940 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.310882    2556 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 20:32:33.311076 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.311046    2556 factory.go:55] Registering systemd factory
Sep 09 20:32:33.311076 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.311060    2556 factory.go:223] Registration of the systemd container factory successfully
Sep 09 20:32:33.311282 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.311255    2556 factory.go:153] Registering CRI-O factory
Sep 09 20:32:33.311282 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.311273    2556 factory.go:223] Registration of the crio container factory successfully
Sep 09 20:32:33.311431 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.311325    2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 20:32:33.311431 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.311344    2556 factory.go:103] Registering Raw factory
Sep 09 20:32:33.311431 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.311358    2556 manager.go:1196] Started watching for new ooms in manager
Sep 09 20:32:33.311763 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.311751    2556 manager.go:349] Starting recovery of all containers
Sep 09 20:32:33.312726 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.312700    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 20:32:33.312821 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.312799    2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-117.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 20:32:33.316338 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.316185    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s68wp"
Sep 09 20:32:33.324034 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.324017    2556 manager.go:354] Recovery completed
Sep 09 20:32:33.325962 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.325944    2556 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 09 20:32:33.326369 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.326340    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s68wp"
Sep 09 20:32:33.329990 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.329899    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:32:33.333084 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.333063    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:32:33.333157 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.333102    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:32:33.333157 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.333120    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:32:33.333696 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.333677    2556 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 20:32:33.333696 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.333694    2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 20:32:33.333844 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.333704    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:32:33.333844 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.333711    2556 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:32:33.334367 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.334352    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:32:33.334423 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.334383    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:32:33.334423 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.334397    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:32:33.343440 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.343377    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-117.ec2.internal.18d3c13f0095007c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-117.ec2.internal,UID:ip-10-0-135-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-117.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-117.ec2.internal,},FirstTimestamp:2026-09-09 20:32:33.333084284 +0000 UTC m=+0.339363304,LastTimestamp:2026-09-09 20:32:33.333084284 +0000 UTC m=+0.339363304,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-117.ec2.internal,}"
Sep 09 20:32:33.347021 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.346963    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-117.ec2.internal.18d3c13f009569a8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-117.ec2.internal,UID:ip-10-0-135-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-117.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-117.ec2.internal,},FirstTimestamp:2026-09-09 20:32:33.333111208 +0000 UTC m=+0.339390223,LastTimestamp:2026-09-09 20:32:33.333111208 +0000 UTC m=+0.339390223,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-117.ec2.internal,}"
Sep 09 20:32:33.351132 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.351079    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-117.ec2.internal.18d3c13f0095a2a8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-117.ec2.internal,UID:ip-10-0-135-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-117.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-117.ec2.internal,},FirstTimestamp:2026-09-09 20:32:33.3331258 +0000 UTC m=+0.339404814,LastTimestamp:2026-09-09 20:32:33.3331258 +0000 UTC m=+0.339404814,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-117.ec2.internal,}"
Sep 09 20:32:33.360345 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.360290    2556 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-117.ec2.internal.18d3c13f0095007c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-117.ec2.internal.18d3c13f0095007c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-117.ec2.internal,UID:ip-10-0-135-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-117.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-117.ec2.internal,},FirstTimestamp:2026-09-09 20:32:33.333084284 +0000 UTC m=+0.339363304,LastTimestamp:2026-09-09 20:32:33.334367873 +0000 UTC m=+0.340646889,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-117.ec2.internal,}"
Sep 09 20:32:33.381346 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.364156    2556 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-117.ec2.internal.18d3c13f009569a8\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-117.ec2.internal.18d3c13f009569a8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-117.ec2.internal,UID:ip-10-0-135-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-117.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-117.ec2.internal,},FirstTimestamp:2026-09-09 20:32:33.333111208 +0000 UTC m=+0.339390223,LastTimestamp:2026-09-09 20:32:33.334389088 +0000 UTC m=+0.340668103,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-117.ec2.internal,}"
Sep 09 20:32:33.381346 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.372975    2556 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-117.ec2.internal.18d3c13f0095a2a8\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-117.ec2.internal.18d3c13f0095a2a8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-117.ec2.internal,UID:ip-10-0-135-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-117.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-117.ec2.internal,},FirstTimestamp:2026-09-09 20:32:33.3331258 +0000 UTC m=+0.339404814,LastTimestamp:2026-09-09 20:32:33.334402754 +0000 UTC m=+0.340681770,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-117.ec2.internal,}"
Sep 09 20:32:33.411089 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.411073    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-117.ec2.internal\" not found"
Sep 09 20:32:33.473798 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.473780    2556 policy_none.go:49] "None policy: Start"
Sep 09 20:32:33.473865 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.473804    2556 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 20:32:33.473865 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.473819    2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 20:32:33.475732 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.475719    2556 policy_none.go:47] "Start"
Sep 09 20:32:33.506066 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.506050    2556 manager.go:337] "Starting Device Plugin manager"
Sep 09 20:32:33.506155 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.506082    2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 20:32:33.506155 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.506092    2556 server.go:87] "Starting device plugin registration server"
Sep 09 20:32:33.506333 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.506322    2556 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 20:32:33.506368 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.506335    2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 20:32:33.506453 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.506438    2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 20:32:33.506522 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.506515    2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 20:32:33.506572 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.506524    2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 20:32:33.507019 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.507004    2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 20:32:33.507105 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.507052    2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-117.ec2.internal\" not found"
Sep 09 20:32:33.522548 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.522525    2556 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-117.ec2.internal\" not found" node="ip-10-0-135-117.ec2.internal"
Sep 09 20:32:33.558241 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.558223    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 20:32:33.559320 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.559302    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 20:32:33.559375 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.559361    2556 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 20:32:33.559422 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.559385    2556 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 20:32:33.559471 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.559461    2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 20:32:33.562121 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.562105    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:32:33.607337 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.607323    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:32:33.609930 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.609916    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:32:33.610003 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.609948    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:32:33.610003 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.609963    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:32:33.610003 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.609985    2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-117.ec2.internal"
Sep 09 20:32:33.617913 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.617897    2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-117.ec2.internal"
Sep 09 20:32:33.617978 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.617915    2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-117.ec2.internal\": node \"ip-10-0-135-117.ec2.internal\" not found"
Sep 09 20:32:33.631745 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.631724    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-117.ec2.internal\" not found"
Sep 09 20:32:33.659573 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.659552    2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-117.ec2.internal"]
Sep 09 20:32:33.659645 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.659625    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:32:33.660832 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.660819    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:32:33.660886 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.660843    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:32:33.660886 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.660853    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:32:33.662921 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.662910    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:32:33.663032 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.663018    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal"
Sep 09 20:32:33.663072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.663047    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:32:33.663602 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.663573    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:32:33.663602 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.663583    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:32:33.663694 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.663617    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:32:33.663694 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.663617    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:32:33.663694 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.663639    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:32:33.663694 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.663626    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:32:33.665749 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.665734    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-117.ec2.internal"
Sep 09 20:32:33.665810 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.665761    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:32:33.666411 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.666389    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:32:33.666411 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.666413    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:32:33.666511 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.666423    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:32:33.693147 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.693124    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-117.ec2.internal\" not found" node="ip-10-0-135-117.ec2.internal"
Sep 09 20:32:33.697412 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.697397    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-117.ec2.internal\" not found" node="ip-10-0-135-117.ec2.internal"
Sep 09 20:32:33.713481 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.713460    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3508d0a07aa0fd49c57a7f8e3fd46334-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal\" (UID: \"3508d0a07aa0fd49c57a7f8e3fd46334\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal"
Sep 09 20:32:33.713548 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.713483    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3508d0a07aa0fd49c57a7f8e3fd46334-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal\" (UID: \"3508d0a07aa0fd49c57a7f8e3fd46334\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal"
Sep 09 20:32:33.713548 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.713499    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4a08d7fc3d09b5f1580b5cd035f70a19-config\") pod \"kube-apiserver-proxy-ip-10-0-135-117.ec2.internal\" (UID: \"4a08d7fc3d09b5f1580b5cd035f70a19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-117.ec2.internal"
Sep 09 20:32:33.732034 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.732015    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-117.ec2.internal\" not found"
Sep 09 20:32:33.814055 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.814001    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3508d0a07aa0fd49c57a7f8e3fd46334-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal\" (UID: \"3508d0a07aa0fd49c57a7f8e3fd46334\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal"
Sep 09 20:32:33.814055 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.814031    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3508d0a07aa0fd49c57a7f8e3fd46334-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal\" (UID: \"3508d0a07aa0fd49c57a7f8e3fd46334\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal"
Sep 09 20:32:33.814055 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.814053    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4a08d7fc3d09b5f1580b5cd035f70a19-config\") pod \"kube-apiserver-proxy-ip-10-0-135-117.ec2.internal\" (UID: \"4a08d7fc3d09b5f1580b5cd035f70a19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-117.ec2.internal"
Sep 09 20:32:33.814208 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.814118    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3508d0a07aa0fd49c57a7f8e3fd46334-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal\" (UID: \"3508d0a07aa0fd49c57a7f8e3fd46334\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal"
Sep 09 20:32:33.814208 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.814124    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3508d0a07aa0fd49c57a7f8e3fd46334-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal\" (UID: \"3508d0a07aa0fd49c57a7f8e3fd46334\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal"
Sep 09 20:32:33.814208 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.814193    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4a08d7fc3d09b5f1580b5cd035f70a19-config\") pod \"kube-apiserver-proxy-ip-10-0-135-117.ec2.internal\" (UID: \"4a08d7fc3d09b5f1580b5cd035f70a19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-117.ec2.internal"
Sep 09 20:32:33.832069 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.832047    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-117.ec2.internal\" not found"
Sep 09 20:32:33.932497 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:33.932477    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-117.ec2.internal\" not found"
Sep 09 20:32:33.995662 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:33.995638    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal"
Sep 09 20:32:34.000692 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:34.000667    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-117.ec2.internal"
Sep 09 20:32:34.033425 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:34.033394    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-117.ec2.internal\" not found"
Sep 09 20:32:34.133962 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:34.133910    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-117.ec2.internal\" not found"
Sep 09 20:32:34.219657 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:34.219640    2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:32:34.234661 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:34.234646    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-117.ec2.internal\" not found"
Sep 09 20:32:34.250812 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:34.250793    2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 20:32:34.250912 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:34.250896    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:32:34.250945 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:34.250932    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:32:34.336837 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:34.336809    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-117.ec2.internal\" not found"
Sep 09 20:32:35.682307 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:35.682275    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:32:35.683127 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:35.683110    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:32:35.683204 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:35.683140    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:32:35.683204 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:35.683152    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:32:35.763449 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:35.763430    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-117.ec2.internal\" not found"
Sep 09 20:32:35.801602 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:35.801573    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:27:34 +0000 UTC" deadline="2028-06-11 13:35:17.941306818 +0000 UTC"
Sep 09 20:32:35.801602 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:35.801598    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15377h2m42.139712085s"
Sep 09 20:32:35.815588 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:35.815572    2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:32:35.837721 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:35.837704    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal"
Sep 09 20:32:35.848822 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:35.848804    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:32:35.851587 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:35.851576    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-117.ec2.internal"
Sep 09 20:32:35.860091 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:35.860076    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:32:36.061665 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:36.061638    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a08d7fc3d09b5f1580b5cd035f70a19.slice/crio-0ed22847f5080b9cc2d10b4f66bd84622001b04aca01b1e1e4a4ad0bc785d3a9 WatchSource:0}: Error finding container 0ed22847f5080b9cc2d10b4f66bd84622001b04aca01b1e1e4a4ad0bc785d3a9: Status 404 returned error can't find the container with id 0ed22847f5080b9cc2d10b4f66bd84622001b04aca01b1e1e4a4ad0bc785d3a9
Sep 09 20:32:36.061960 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:36.061945    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3508d0a07aa0fd49c57a7f8e3fd46334.slice/crio-36412708256f6f81261cfd3112d323cf6033b62d1fbe6ade506630241418d46a WatchSource:0}: Error finding container 36412708256f6f81261cfd3112d323cf6033b62d1fbe6ade506630241418d46a: Status 404 returned error can't find the container with id 36412708256f6f81261cfd3112d323cf6033b62d1fbe6ade506630241418d46a
Sep 09 20:32:36.066474 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.066458    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:32:36.146102 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.146074    2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:32:36.415471 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.415417    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:32:36.615799 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.615782    2556 apiserver.go:52] "Watching apiserver"
Sep 09 20:32:36.623773 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.623757    2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 20:32:36.625573 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.625554    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-w8v2w","openshift-multus/network-metrics-daemon-bwnpx","openshift-network-diagnostics/network-check-target-2zpmx","openshift-network-operator/iptables-alerter-9rvmd","kube-system/global-pull-secret-syncer-q2zw5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n","openshift-dns/node-resolver-bmnnt","openshift-image-registry/node-ca-kj4jd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal","openshift-multus/multus-additional-cni-plugins-m86gm","openshift-multus/multus-wkmrq","openshift-ovn-kubernetes/ovnkube-node-25h4s","kube-system/konnectivity-agent-jr87x","kube-system/kube-apiserver-proxy-ip-10-0-135-117.ec2.internal"]
Sep 09 20:32:36.629326 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.629299    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.631407 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.631383    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:32:36.631482 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:36.631457    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwnpx" podUID="73ad4e42-0fc1-4d62-afcf-596c76c4ea87"
Sep 09 20:32:36.632051 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.632030    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:32:36.632133 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.632030    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6zh7f\""
Sep 09 20:32:36.632133 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.632062    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 20:32:36.633376 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.633357    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:32:36.633482 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:36.633411    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zpmx" podUID="14c3f4cb-c792-4945-bd65-932ed6a27aa9"
Sep 09 20:32:36.635447 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.635427    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jr87x"
Sep 09 20:32:36.637674 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.637652    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.637864 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.637702    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 20:32:36.639606 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.639441    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 20:32:36.639606 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.639456    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pgp6n\""
Sep 09 20:32:36.639757 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.639626    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 20:32:36.640315 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.640287    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 20:32:36.641473 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.641455    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kj4jd"
Sep 09 20:32:36.641570 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.641531    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t6zxd\""
Sep 09 20:32:36.641633 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.641586    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 20:32:36.641633 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.641587    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 20:32:36.642861 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.642841    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 20:32:36.642973 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.642921    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 20:32:36.642973 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.642961    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 20:32:36.643874 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.643854    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bmnnt"
Sep 09 20:32:36.644620 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.644603    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 20:32:36.645196 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.645181    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 20:32:36.646207 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.646193    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n"
Sep 09 20:32:36.646689 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.646657    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 20:32:36.646870 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.646854    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-host-run-ovn-kubernetes\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.646958 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.646886    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrftj\" (UniqueName: \"kubernetes.io/projected/33ba58cf-b5d2-4b22-83d2-ef5e54576651-kube-api-access-rrftj\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.646958 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.646925    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-lib-modules\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.646958 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.646948    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-host\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.647104 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.646971    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llb42\" (UniqueName: \"kubernetes.io/projected/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-kube-api-access-llb42\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.647104 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647005    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj47h\" (UniqueName: \"kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h\") pod \"network-check-target-2zpmx\" (UID: \"14c3f4cb-c792-4945-bd65-932ed6a27aa9\") " pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:32:36.647104 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647047    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-host-slash\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.647104 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647058    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gnvbg\""
Sep 09 20:32:36.647104 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647072    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-etc-sysctl-d\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.647104 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647086    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-etc-systemd\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.647104 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647086    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 20:32:36.647384 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647117    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-sys\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.647384 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647088    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-grplz\""
Sep 09 20:32:36.647384 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647148    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7241694e-5bba-4400-9eb9-771af5989280-agent-certs\") pod \"konnectivity-agent-jr87x\" (UID: \"7241694e-5bba-4400-9eb9-771af5989280\") " pod="kube-system/konnectivity-agent-jr87x"
Sep 09 20:32:36.647384 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647188    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-run-systemd\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.647384 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647227    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-host-cni-netd\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.647384 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647266    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.647384 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647294    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-tmp\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.647384 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647312    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-run-openvswitch\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.647384 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647333    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-log-socket\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.647384 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647368    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/33ba58cf-b5d2-4b22-83d2-ef5e54576651-ovnkube-config\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.647384 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647387    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/33ba58cf-b5d2-4b22-83d2-ef5e54576651-env-overrides\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.648326 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647402    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-host-cni-bin\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.648326 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647434    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/33ba58cf-b5d2-4b22-83d2-ef5e54576651-ovn-node-metrics-cert\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.648326 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647450    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-run\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.648326 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647463    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-etc-tuned\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.648326 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647482    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-host-run-netns\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.648326 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647504    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-run-ovn\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.648326 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647523    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/33ba58cf-b5d2-4b22-83d2-ef5e54576651-ovnkube-script-lib\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.648326 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647542    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-etc-modprobe-d\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.648326 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647557    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9tqq\" (UniqueName: \"kubernetes.io/projected/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-kube-api-access-x9tqq\") pod \"network-metrics-daemon-bwnpx\" (UID: \"73ad4e42-0fc1-4d62-afcf-596c76c4ea87\") " pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:32:36.648326 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647580    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7241694e-5bba-4400-9eb9-771af5989280-konnectivity-ca\") pod \"konnectivity-agent-jr87x\" (UID: \"7241694e-5bba-4400-9eb9-771af5989280\") " pod="kube-system/konnectivity-agent-jr87x"
Sep 09 20:32:36.648326 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647606    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-var-lib-openvswitch\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.648326 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647660    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-etc-kubernetes\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.648326 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647699    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs\") pod \"network-metrics-daemon-bwnpx\" (UID: \"73ad4e42-0fc1-4d62-afcf-596c76c4ea87\") " pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:32:36.648326 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647745    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-host-kubelet\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.648326 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647773    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-systemd-units\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.648326 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647807    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-etc-openvswitch\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.649303 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647831    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-node-log\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.649303 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647852    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-etc-sysconfig\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.649303 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647873    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-etc-sysctl-conf\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.649303 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.647925    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-var-lib-kubelet\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.649303 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.648126    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 20:32:36.649303 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.648621    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m86gm"
Sep 09 20:32:36.650388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.650364    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 20:32:36.650800 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.650649    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ngvmm\""
Sep 09 20:32:36.650887 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.650871    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 20:32:36.651077 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.651055    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.653373 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.653354    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:36.653467 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:36.653419    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zw5" podUID="61bb5c63-cbb7-4186-9c23-5706dba8cec2"
Sep 09 20:32:36.654523 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.654397    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 20:32:36.654523 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.654501    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 20:32:36.654664 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.654648    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 20:32:36.654723 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.654703    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 20:32:36.654723 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.654713    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:32:36.655796 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.655775    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9rvmd"
Sep 09 20:32:36.655796 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.655790    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 20:32:36.655953 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.655813    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 20:32:36.655953 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.655825    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jls7v\""
Sep 09 20:32:36.655953 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.655782    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hkwnw\""
Sep 09 20:32:36.658920 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.658888    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:32:36.659007 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.658922    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 20:32:36.659064 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.659004    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2r7j6\""
Sep 09 20:32:36.659274 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.659259    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 20:32:36.662327 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.662309    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 20:32:36.721015 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.720993    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-g7s7f"
Sep 09 20:32:36.727680 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.727625    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g7s7f"
Sep 09 20:32:36.738850 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.738000    2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 20:32:36.748155 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748130    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-run-systemd\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.748250 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748165    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-host-cni-netd\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.748250 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748193    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-multus-cni-dir\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.748329 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748257    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/86f6ae47-3710-4293-ba2b-73982bcf0e84-sys-fs\") pod \"aws-ebs-csi-driver-node-rmn7n\" (UID: \"86f6ae47-3710-4293-ba2b-73982bcf0e84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n"
Sep 09 20:32:36.748329 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748291    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-host-cni-netd\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.748329 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748305    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc6xq\" (UniqueName: \"kubernetes.io/projected/21bf48f4-01d2-4e62-af48-52956be58f15-kube-api-access-jc6xq\") pod \"multus-additional-cni-plugins-m86gm\" (UID: \"21bf48f4-01d2-4e62-af48-52956be58f15\") " pod="openshift-multus/multus-additional-cni-plugins-m86gm"
Sep 09 20:32:36.748452 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748260    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-run-systemd\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.748452 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748345    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-run-openvswitch\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.748452 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748372    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-log-socket\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.748452 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748401    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/33ba58cf-b5d2-4b22-83d2-ef5e54576651-ovnkube-config\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.748452 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748422    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/33ba58cf-b5d2-4b22-83d2-ef5e54576651-env-overrides\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.748452 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748445    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-host-cni-bin\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.748723 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748455    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-log-socket\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.748723 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748469    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/33ba58cf-b5d2-4b22-83d2-ef5e54576651-ovn-node-metrics-cert\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.748723 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748493    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-run\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.748723 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748499    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-run-openvswitch\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.748723 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748523    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-etc-tuned\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.748723 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748542    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-host-cni-bin\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.748723 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748550    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2ntl\" (UniqueName: \"kubernetes.io/projected/86f6ae47-3710-4293-ba2b-73982bcf0e84-kube-api-access-d2ntl\") pod \"aws-ebs-csi-driver-node-rmn7n\" (UID: \"86f6ae47-3710-4293-ba2b-73982bcf0e84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n"
Sep 09 20:32:36.748723 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748576    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/33ba58cf-b5d2-4b22-83d2-ef5e54576651-ovnkube-script-lib\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.748723 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748599    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-etc-modprobe-d\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.748723 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748622    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/86f6ae47-3710-4293-ba2b-73982bcf0e84-device-dir\") pod \"aws-ebs-csi-driver-node-rmn7n\" (UID: \"86f6ae47-3710-4293-ba2b-73982bcf0e84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n"
Sep 09 20:32:36.748723 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748645    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/21bf48f4-01d2-4e62-af48-52956be58f15-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m86gm\" (UID: \"21bf48f4-01d2-4e62-af48-52956be58f15\") " pod="openshift-multus/multus-additional-cni-plugins-m86gm"
Sep 09 20:32:36.748723 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748668    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-host-var-lib-cni-multus\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.748723 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748694    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs\") pod \"network-metrics-daemon-bwnpx\" (UID: \"73ad4e42-0fc1-4d62-afcf-596c76c4ea87\") " pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:32:36.748723 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748715    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c31d1698-1b9c-451e-995a-ed666693b6a2-serviceca\") pod \"node-ca-kj4jd\" (UID: \"c31d1698-1b9c-451e-995a-ed666693b6a2\") " pod="openshift-image-registry/node-ca-kj4jd"
Sep 09 20:32:36.749388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748745    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-etc-openvswitch\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.749388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748770    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-etc-sysconfig\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.749388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748792    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-etc-sysctl-conf\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.749388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748815    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-multus-socket-dir-parent\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.749388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748841    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5lpw\" (UniqueName: \"kubernetes.io/projected/65da59e2-9681-4904-bd9a-ed612f4d54b9-kube-api-access-j5lpw\") pod \"iptables-alerter-9rvmd\" (UID: \"65da59e2-9681-4904-bd9a-ed612f4d54b9\") " pod="openshift-network-operator/iptables-alerter-9rvmd"
Sep 09 20:32:36.749388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748865    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21bf48f4-01d2-4e62-af48-52956be58f15-system-cni-dir\") pod \"multus-additional-cni-plugins-m86gm\" (UID: \"21bf48f4-01d2-4e62-af48-52956be58f15\") " pod="openshift-multus/multus-additional-cni-plugins-m86gm"
Sep 09 20:32:36.749388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748890    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-host-run-ovn-kubernetes\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.749388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748926    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-host\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.749388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748949    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-host-var-lib-kubelet\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.749388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.748978    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/86f6ae47-3710-4293-ba2b-73982bcf0e84-socket-dir\") pod \"aws-ebs-csi-driver-node-rmn7n\" (UID: \"86f6ae47-3710-4293-ba2b-73982bcf0e84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n"
Sep 09 20:32:36.749388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749002    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/86f6ae47-3710-4293-ba2b-73982bcf0e84-registration-dir\") pod \"aws-ebs-csi-driver-node-rmn7n\" (UID: \"86f6ae47-3710-4293-ba2b-73982bcf0e84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n"
Sep 09 20:32:36.749388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749025    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-etc-systemd\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.749388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749036    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/33ba58cf-b5d2-4b22-83d2-ef5e54576651-env-overrides\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.749388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749048    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-sys\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.749388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749071    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7241694e-5bba-4400-9eb9-771af5989280-agent-certs\") pod \"konnectivity-agent-jr87x\" (UID: \"7241694e-5bba-4400-9eb9-771af5989280\") " pod="kube-system/konnectivity-agent-jr87x"
Sep 09 20:32:36.749388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749098    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-multus-daemon-config\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.749388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749116    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-run\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.750182 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749120    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/61bb5c63-cbb7-4186-9c23-5706dba8cec2-dbus\") pod \"global-pull-secret-syncer-q2zw5\" (UID: \"61bb5c63-cbb7-4186-9c23-5706dba8cec2\") " pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:36.750182 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749156    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.750182 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749183    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-tmp\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.750182 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749210    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9khkh\" (UniqueName: \"kubernetes.io/projected/c31d1698-1b9c-451e-995a-ed666693b6a2-kube-api-access-9khkh\") pod \"node-ca-kj4jd\" (UID: \"c31d1698-1b9c-451e-995a-ed666693b6a2\") " pod="openshift-image-registry/node-ca-kj4jd"
Sep 09 20:32:36.750182 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749216    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/33ba58cf-b5d2-4b22-83d2-ef5e54576651-ovnkube-config\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.750182 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749235    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret\") pod \"global-pull-secret-syncer-q2zw5\" (UID: \"61bb5c63-cbb7-4186-9c23-5706dba8cec2\") " pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:36.750182 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749258    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c31d1698-1b9c-451e-995a-ed666693b6a2-host\") pod \"node-ca-kj4jd\" (UID: \"c31d1698-1b9c-451e-995a-ed666693b6a2\") " pod="openshift-image-registry/node-ca-kj4jd"
Sep 09 20:32:36.750182 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749280    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-system-cni-dir\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.750182 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749303    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-host-run-netns\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.750182 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749309    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/33ba58cf-b5d2-4b22-83d2-ef5e54576651-ovnkube-script-lib\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.750182 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749326    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-multus-conf-dir\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.750182 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749327    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-etc-sysconfig\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.750182 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749425    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-etc-sysctl-conf\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.750182 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749429    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-host-run-ovn-kubernetes\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.750182 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749444    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-etc-modprobe-d\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.750182 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749475    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-host\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.750182 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749494    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.750998 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749501    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-etc-systemd\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.750998 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749527    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61416974-f481-4e36-ab4a-2be050c06afe-tmp-dir\") pod \"node-resolver-bmnnt\" (UID: \"61416974-f481-4e36-ab4a-2be050c06afe\") " pod="openshift-dns/node-resolver-bmnnt"
Sep 09 20:32:36.750998 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749542    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/86f6ae47-3710-4293-ba2b-73982bcf0e84-etc-selinux\") pod \"aws-ebs-csi-driver-node-rmn7n\" (UID: \"86f6ae47-3710-4293-ba2b-73982bcf0e84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n"
Sep 09 20:32:36.750998 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749556    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21bf48f4-01d2-4e62-af48-52956be58f15-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m86gm\" (UID: \"21bf48f4-01d2-4e62-af48-52956be58f15\") " pod="openshift-multus/multus-additional-cni-plugins-m86gm"
Sep 09 20:32:36.750998 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749571    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/21bf48f4-01d2-4e62-af48-52956be58f15-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m86gm\" (UID: \"21bf48f4-01d2-4e62-af48-52956be58f15\") " pod="openshift-multus/multus-additional-cni-plugins-m86gm"
Sep 09 20:32:36.750998 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749594    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-host-run-netns\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.750998 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749604    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-sys\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.750998 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749609    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-run-ovn\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.750998 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749626    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-run-ovn\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.750998 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749646    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9tqq\" (UniqueName: \"kubernetes.io/projected/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-kube-api-access-x9tqq\") pod \"network-metrics-daemon-bwnpx\" (UID: \"73ad4e42-0fc1-4d62-afcf-596c76c4ea87\") " pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:32:36.750998 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749659    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-host-run-netns\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.750998 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749572    2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 20:32:36.750998 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749281    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-etc-openvswitch\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.750998 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:36.749235    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:32:36.750998 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:36.749775    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs podName:73ad4e42-0fc1-4d62-afcf-596c76c4ea87 nodeName:}" failed. No retries permitted until 2026-09-09 20:32:37.249755708 +0000 UTC m=+2.929715998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs") pod "network-metrics-daemon-bwnpx" (UID: "73ad4e42-0fc1-4d62-afcf-596c76c4ea87") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:32:36.750998 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749665    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7241694e-5bba-4400-9eb9-771af5989280-konnectivity-ca\") pod \"konnectivity-agent-jr87x\" (UID: \"7241694e-5bba-4400-9eb9-771af5989280\") " pod="kube-system/konnectivity-agent-jr87x"
Sep 09 20:32:36.750998 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749953    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-cnibin\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.751556 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.749983    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-host-run-k8s-cni-cncf-io\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.751556 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750010    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-host-run-multus-certs\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.751556 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750048    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-hostroot\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.751556 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750076    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-var-lib-openvswitch\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.751556 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750096    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-etc-kubernetes\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.751556 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750115    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-cni-binary-copy\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.751556 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750119    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7241694e-5bba-4400-9eb9-771af5989280-konnectivity-ca\") pod \"konnectivity-agent-jr87x\" (UID: \"7241694e-5bba-4400-9eb9-771af5989280\") " pod="kube-system/konnectivity-agent-jr87x"
Sep 09 20:32:36.751556 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750135    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-host-var-lib-cni-bin\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.751556 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750152    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-var-lib-openvswitch\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.751556 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750158    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dphwt\" (UniqueName: \"kubernetes.io/projected/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-kube-api-access-dphwt\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.751556 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750196    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65da59e2-9681-4904-bd9a-ed612f4d54b9-host-slash\") pod \"iptables-alerter-9rvmd\" (UID: \"65da59e2-9681-4904-bd9a-ed612f4d54b9\") " pod="openshift-network-operator/iptables-alerter-9rvmd"
Sep 09 20:32:36.751556 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750221    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21bf48f4-01d2-4e62-af48-52956be58f15-cni-binary-copy\") pod \"multus-additional-cni-plugins-m86gm\" (UID: \"21bf48f4-01d2-4e62-af48-52956be58f15\") " pod="openshift-multus/multus-additional-cni-plugins-m86gm"
Sep 09 20:32:36.751556 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750217    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-etc-kubernetes\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.751556 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750250    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-host-kubelet\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.751556 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750275    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-systemd-units\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.751556 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750298    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-node-log\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.751556 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750321    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-var-lib-kubelet\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.752321 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750360    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-etc-kubernetes\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.752321 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750370    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-systemd-units\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.752321 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750385    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65da59e2-9681-4904-bd9a-ed612f4d54b9-iptables-alerter-script\") pod \"iptables-alerter-9rvmd\" (UID: \"65da59e2-9681-4904-bd9a-ed612f4d54b9\") " pod="openshift-network-operator/iptables-alerter-9rvmd"
Sep 09 20:32:36.752321 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750408    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/61416974-f481-4e36-ab4a-2be050c06afe-hosts-file\") pod \"node-resolver-bmnnt\" (UID: \"61416974-f481-4e36-ab4a-2be050c06afe\") " pod="openshift-dns/node-resolver-bmnnt"
Sep 09 20:32:36.752321 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750423    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-host-kubelet\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.752321 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750431    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21bf48f4-01d2-4e62-af48-52956be58f15-cnibin\") pod \"multus-additional-cni-plugins-m86gm\" (UID: \"21bf48f4-01d2-4e62-af48-52956be58f15\") " pod="openshift-multus/multus-additional-cni-plugins-m86gm"
Sep 09 20:32:36.752321 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750457    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrftj\" (UniqueName: \"kubernetes.io/projected/33ba58cf-b5d2-4b22-83d2-ef5e54576651-kube-api-access-rrftj\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.752321 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750473    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-node-log\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.752321 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750480    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-lib-modules\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.752321 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750503    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llb42\" (UniqueName: \"kubernetes.io/projected/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-kube-api-access-llb42\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.752321 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750520    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj47h\" (UniqueName: \"kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h\") pod \"network-check-target-2zpmx\" (UID: \"14c3f4cb-c792-4945-bd65-932ed6a27aa9\") " pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:32:36.752321 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750526    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-var-lib-kubelet\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.752321 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750536    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhqkt\" (UniqueName: \"kubernetes.io/projected/61416974-f481-4e36-ab4a-2be050c06afe-kube-api-access-fhqkt\") pod \"node-resolver-bmnnt\" (UID: \"61416974-f481-4e36-ab4a-2be050c06afe\") " pod="openshift-dns/node-resolver-bmnnt"
Sep 09 20:32:36.752321 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750560    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-host-slash\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.752321 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750583    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-etc-sysctl-d\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.752321 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750599    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/61bb5c63-cbb7-4186-9c23-5706dba8cec2-kubelet-config\") pod \"global-pull-secret-syncer-q2zw5\" (UID: \"61bb5c63-cbb7-4186-9c23-5706dba8cec2\") " pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:36.752321 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750612    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/86f6ae47-3710-4293-ba2b-73982bcf0e84-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rmn7n\" (UID: \"86f6ae47-3710-4293-ba2b-73982bcf0e84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n"
Sep 09 20:32:36.753139 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750745    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-lib-modules\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.753139 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750794    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33ba58cf-b5d2-4b22-83d2-ef5e54576651-host-slash\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.753139 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.750884    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-etc-sysctl-d\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.753613 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.753590    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/33ba58cf-b5d2-4b22-83d2-ef5e54576651-ovn-node-metrics-cert\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.753693 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.753647    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-tmp\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.753844 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.753828    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-etc-tuned\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.755591 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.755572    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7241694e-5bba-4400-9eb9-771af5989280-agent-certs\") pod \"konnectivity-agent-jr87x\" (UID: \"7241694e-5bba-4400-9eb9-771af5989280\") " pod="kube-system/konnectivity-agent-jr87x"
Sep 09 20:32:36.761104 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:36.760736    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:32:36.761104 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:36.760761    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:32:36.761104 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:36.760775    2556 projected.go:196] Error preparing data for projected volume kube-api-access-nj47h for pod openshift-network-diagnostics/network-check-target-2zpmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:32:36.761104 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:36.760852    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h podName:14c3f4cb-c792-4945-bd65-932ed6a27aa9 nodeName:}" failed. No retries permitted until 2026-09-09 20:32:37.260833889 +0000 UTC m=+2.940794191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nj47h" (UniqueName: "kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h") pod "network-check-target-2zpmx" (UID: "14c3f4cb-c792-4945-bd65-932ed6a27aa9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:32:36.761422 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.761284    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9tqq\" (UniqueName: \"kubernetes.io/projected/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-kube-api-access-x9tqq\") pod \"network-metrics-daemon-bwnpx\" (UID: \"73ad4e42-0fc1-4d62-afcf-596c76c4ea87\") " pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:32:36.763214 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.763180    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrftj\" (UniqueName: \"kubernetes.io/projected/33ba58cf-b5d2-4b22-83d2-ef5e54576651-kube-api-access-rrftj\") pod \"ovnkube-node-25h4s\" (UID: \"33ba58cf-b5d2-4b22-83d2-ef5e54576651\") " pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.763430 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.763411    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llb42\" (UniqueName: \"kubernetes.io/projected/ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40-kube-api-access-llb42\") pod \"tuned-w8v2w\" (UID: \"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40\") " pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.851273 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851247    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9khkh\" (UniqueName: \"kubernetes.io/projected/c31d1698-1b9c-451e-995a-ed666693b6a2-kube-api-access-9khkh\") pod \"node-ca-kj4jd\" (UID: \"c31d1698-1b9c-451e-995a-ed666693b6a2\") " pod="openshift-image-registry/node-ca-kj4jd"
Sep 09 20:32:36.851273 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851276    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret\") pod \"global-pull-secret-syncer-q2zw5\" (UID: \"61bb5c63-cbb7-4186-9c23-5706dba8cec2\") " pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:36.851436 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851299    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c31d1698-1b9c-451e-995a-ed666693b6a2-host\") pod \"node-ca-kj4jd\" (UID: \"c31d1698-1b9c-451e-995a-ed666693b6a2\") " pod="openshift-image-registry/node-ca-kj4jd"
Sep 09 20:32:36.851436 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851327    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-system-cni-dir\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.851436 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851350    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-host-run-netns\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.851436 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851372    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-multus-conf-dir\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.851436 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851393    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61416974-f481-4e36-ab4a-2be050c06afe-tmp-dir\") pod \"node-resolver-bmnnt\" (UID: \"61416974-f481-4e36-ab4a-2be050c06afe\") " pod="openshift-dns/node-resolver-bmnnt"
Sep 09 20:32:36.851436 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851417    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/86f6ae47-3710-4293-ba2b-73982bcf0e84-etc-selinux\") pod \"aws-ebs-csi-driver-node-rmn7n\" (UID: \"86f6ae47-3710-4293-ba2b-73982bcf0e84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n"
Sep 09 20:32:36.851735 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851439    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21bf48f4-01d2-4e62-af48-52956be58f15-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m86gm\" (UID: \"21bf48f4-01d2-4e62-af48-52956be58f15\") " pod="openshift-multus/multus-additional-cni-plugins-m86gm"
Sep 09 20:32:36.851735 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851467    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/21bf48f4-01d2-4e62-af48-52956be58f15-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m86gm\" (UID: \"21bf48f4-01d2-4e62-af48-52956be58f15\") " pod="openshift-multus/multus-additional-cni-plugins-m86gm"
Sep 09 20:32:36.851735 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851495    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-cnibin\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.851735 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851520    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-host-run-k8s-cni-cncf-io\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.851735 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851525    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-host-run-netns\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.851735 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851545    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-host-run-multus-certs\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.851735 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851569    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-hostroot\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.851735 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851600    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-cni-binary-copy\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.851735 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:36.851615    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:32:36.851735 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851621    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-host-var-lib-cni-bin\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.851735 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851644    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dphwt\" (UniqueName: \"kubernetes.io/projected/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-kube-api-access-dphwt\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.851735 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:36.851666    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret podName:61bb5c63-cbb7-4186-9c23-5706dba8cec2 nodeName:}" failed. No retries permitted until 2026-09-09 20:32:37.351647056 +0000 UTC m=+3.031607360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret") pod "global-pull-secret-syncer-q2zw5" (UID: "61bb5c63-cbb7-4186-9c23-5706dba8cec2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:32:36.851735 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851696    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65da59e2-9681-4904-bd9a-ed612f4d54b9-host-slash\") pod \"iptables-alerter-9rvmd\" (UID: \"65da59e2-9681-4904-bd9a-ed612f4d54b9\") " pod="openshift-network-operator/iptables-alerter-9rvmd"
Sep 09 20:32:36.851735 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851717    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-multus-conf-dir\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.851735 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851724    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21bf48f4-01d2-4e62-af48-52956be58f15-cni-binary-copy\") pod \"multus-additional-cni-plugins-m86gm\" (UID: \"21bf48f4-01d2-4e62-af48-52956be58f15\") " pod="openshift-multus/multus-additional-cni-plugins-m86gm"
Sep 09 20:32:36.852471 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851754    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-etc-kubernetes\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.852471 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851782    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65da59e2-9681-4904-bd9a-ed612f4d54b9-iptables-alerter-script\") pod \"iptables-alerter-9rvmd\" (UID: \"65da59e2-9681-4904-bd9a-ed612f4d54b9\") " pod="openshift-network-operator/iptables-alerter-9rvmd"
Sep 09 20:32:36.852471 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851805    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/61416974-f481-4e36-ab4a-2be050c06afe-hosts-file\") pod \"node-resolver-bmnnt\" (UID: \"61416974-f481-4e36-ab4a-2be050c06afe\") " pod="openshift-dns/node-resolver-bmnnt"
Sep 09 20:32:36.852471 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851831    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21bf48f4-01d2-4e62-af48-52956be58f15-cnibin\") pod \"multus-additional-cni-plugins-m86gm\" (UID: \"21bf48f4-01d2-4e62-af48-52956be58f15\") " pod="openshift-multus/multus-additional-cni-plugins-m86gm"
Sep 09 20:32:36.852471 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851871    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhqkt\" (UniqueName: \"kubernetes.io/projected/61416974-f481-4e36-ab4a-2be050c06afe-kube-api-access-fhqkt\") pod \"node-resolver-bmnnt\" (UID: \"61416974-f481-4e36-ab4a-2be050c06afe\") " pod="openshift-dns/node-resolver-bmnnt"
Sep 09 20:32:36.852471 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851897    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/61bb5c63-cbb7-4186-9c23-5706dba8cec2-kubelet-config\") pod \"global-pull-secret-syncer-q2zw5\" (UID: \"61bb5c63-cbb7-4186-9c23-5706dba8cec2\") " pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:36.852471 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851942    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/86f6ae47-3710-4293-ba2b-73982bcf0e84-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rmn7n\" (UID: \"86f6ae47-3710-4293-ba2b-73982bcf0e84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n"
Sep 09 20:32:36.852471 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851970    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-multus-cni-dir\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.852471 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851993    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/86f6ae47-3710-4293-ba2b-73982bcf0e84-sys-fs\") pod \"aws-ebs-csi-driver-node-rmn7n\" (UID: \"86f6ae47-3710-4293-ba2b-73982bcf0e84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n"
Sep 09 20:32:36.852471 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.851998    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61416974-f481-4e36-ab4a-2be050c06afe-tmp-dir\") pod \"node-resolver-bmnnt\" (UID: \"61416974-f481-4e36-ab4a-2be050c06afe\") " pod="openshift-dns/node-resolver-bmnnt"
Sep 09 20:32:36.852471 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852017    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc6xq\" (UniqueName: \"kubernetes.io/projected/21bf48f4-01d2-4e62-af48-52956be58f15-kube-api-access-jc6xq\") pod \"multus-additional-cni-plugins-m86gm\" (UID: \"21bf48f4-01d2-4e62-af48-52956be58f15\") " pod="openshift-multus/multus-additional-cni-plugins-m86gm"
Sep 09 20:32:36.852471 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852055    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2ntl\" (UniqueName: \"kubernetes.io/projected/86f6ae47-3710-4293-ba2b-73982bcf0e84-kube-api-access-d2ntl\") pod \"aws-ebs-csi-driver-node-rmn7n\" (UID: \"86f6ae47-3710-4293-ba2b-73982bcf0e84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n"
Sep 09 20:32:36.852471 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852063    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/86f6ae47-3710-4293-ba2b-73982bcf0e84-etc-selinux\") pod \"aws-ebs-csi-driver-node-rmn7n\" (UID: \"86f6ae47-3710-4293-ba2b-73982bcf0e84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n"
Sep 09 20:32:36.852471 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852084    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/86f6ae47-3710-4293-ba2b-73982bcf0e84-device-dir\") pod \"aws-ebs-csi-driver-node-rmn7n\" (UID: \"86f6ae47-3710-4293-ba2b-73982bcf0e84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n"
Sep 09 20:32:36.852471 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852108    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/21bf48f4-01d2-4e62-af48-52956be58f15-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m86gm\" (UID: \"21bf48f4-01d2-4e62-af48-52956be58f15\") " pod="openshift-multus/multus-additional-cni-plugins-m86gm"
Sep 09 20:32:36.852471 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852134    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-host-var-lib-cni-multus\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.852471 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852170    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c31d1698-1b9c-451e-995a-ed666693b6a2-serviceca\") pod \"node-ca-kj4jd\" (UID: \"c31d1698-1b9c-451e-995a-ed666693b6a2\") " pod="openshift-image-registry/node-ca-kj4jd"
Sep 09 20:32:36.853306 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852189    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21bf48f4-01d2-4e62-af48-52956be58f15-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m86gm\" (UID: \"21bf48f4-01d2-4e62-af48-52956be58f15\") " pod="openshift-multus/multus-additional-cni-plugins-m86gm"
Sep 09 20:32:36.853306 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852198    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-multus-socket-dir-parent\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.853306 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852224    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5lpw\" (UniqueName: \"kubernetes.io/projected/65da59e2-9681-4904-bd9a-ed612f4d54b9-kube-api-access-j5lpw\") pod \"iptables-alerter-9rvmd\" (UID: \"65da59e2-9681-4904-bd9a-ed612f4d54b9\") " pod="openshift-network-operator/iptables-alerter-9rvmd"
Sep 09 20:32:36.853306 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852251    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21bf48f4-01d2-4e62-af48-52956be58f15-system-cni-dir\") pod \"multus-additional-cni-plugins-m86gm\" (UID: \"21bf48f4-01d2-4e62-af48-52956be58f15\") " pod="openshift-multus/multus-additional-cni-plugins-m86gm"
Sep 09 20:32:36.853306 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852278    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-host-var-lib-kubelet\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.853306 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852306    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/86f6ae47-3710-4293-ba2b-73982bcf0e84-socket-dir\") pod \"aws-ebs-csi-driver-node-rmn7n\" (UID: \"86f6ae47-3710-4293-ba2b-73982bcf0e84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n"
Sep 09 20:32:36.853306 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852327    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c31d1698-1b9c-451e-995a-ed666693b6a2-host\") pod \"node-ca-kj4jd\" (UID: \"c31d1698-1b9c-451e-995a-ed666693b6a2\") " pod="openshift-image-registry/node-ca-kj4jd"
Sep 09 20:32:36.853306 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852331    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/86f6ae47-3710-4293-ba2b-73982bcf0e84-registration-dir\") pod \"aws-ebs-csi-driver-node-rmn7n\" (UID: \"86f6ae47-3710-4293-ba2b-73982bcf0e84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n"
Sep 09 20:32:36.853306 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852351    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-host-run-multus-certs\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.853306 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852377    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-multus-daemon-config\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.853306 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852385    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/86f6ae47-3710-4293-ba2b-73982bcf0e84-registration-dir\") pod \"aws-ebs-csi-driver-node-rmn7n\" (UID: \"86f6ae47-3710-4293-ba2b-73982bcf0e84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n"
Sep 09 20:32:36.853306 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852404    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/61bb5c63-cbb7-4186-9c23-5706dba8cec2-dbus\") pod \"global-pull-secret-syncer-q2zw5\" (UID: \"61bb5c63-cbb7-4186-9c23-5706dba8cec2\") " pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:36.853306 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852425    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-cnibin\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.853306 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852435    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-multus-cni-dir\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.853306 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852480    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-host-run-k8s-cni-cncf-io\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.853306 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852588    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/86f6ae47-3710-4293-ba2b-73982bcf0e84-sys-fs\") pod \"aws-ebs-csi-driver-node-rmn7n\" (UID: \"86f6ae47-3710-4293-ba2b-73982bcf0e84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n"
Sep 09 20:32:36.853306 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852686    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/21bf48f4-01d2-4e62-af48-52956be58f15-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m86gm\" (UID: \"21bf48f4-01d2-4e62-af48-52956be58f15\") " pod="openshift-multus/multus-additional-cni-plugins-m86gm"
Sep 09 20:32:36.854033 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852724    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65da59e2-9681-4904-bd9a-ed612f4d54b9-host-slash\") pod \"iptables-alerter-9rvmd\" (UID: \"65da59e2-9681-4904-bd9a-ed612f4d54b9\") " pod="openshift-network-operator/iptables-alerter-9rvmd"
Sep 09 20:32:36.854033 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852740    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-hostroot\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.854033 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.852811    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/86f6ae47-3710-4293-ba2b-73982bcf0e84-device-dir\") pod \"aws-ebs-csi-driver-node-rmn7n\" (UID: \"86f6ae47-3710-4293-ba2b-73982bcf0e84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n"
Sep 09 20:32:36.854033 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.853029    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c31d1698-1b9c-451e-995a-ed666693b6a2-serviceca\") pod \"node-ca-kj4jd\" (UID: \"c31d1698-1b9c-451e-995a-ed666693b6a2\") " pod="openshift-image-registry/node-ca-kj4jd"
Sep 09 20:32:36.854033 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.853100    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-multus-socket-dir-parent\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.854033 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.853204    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21bf48f4-01d2-4e62-af48-52956be58f15-cni-binary-copy\") pod \"multus-additional-cni-plugins-m86gm\" (UID: \"21bf48f4-01d2-4e62-af48-52956be58f15\") " pod="openshift-multus/multus-additional-cni-plugins-m86gm"
Sep 09 20:32:36.854033 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.853224    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-cni-binary-copy\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.854033 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.853257    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-etc-kubernetes\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.854033 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.853323    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21bf48f4-01d2-4e62-af48-52956be58f15-system-cni-dir\") pod \"multus-additional-cni-plugins-m86gm\" (UID: \"21bf48f4-01d2-4e62-af48-52956be58f15\") " pod="openshift-multus/multus-additional-cni-plugins-m86gm"
Sep 09 20:32:36.854033 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.853329    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/21bf48f4-01d2-4e62-af48-52956be58f15-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m86gm\" (UID: \"21bf48f4-01d2-4e62-af48-52956be58f15\") " pod="openshift-multus/multus-additional-cni-plugins-m86gm"
Sep 09 20:32:36.854033 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.853398    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/86f6ae47-3710-4293-ba2b-73982bcf0e84-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rmn7n\" (UID: \"86f6ae47-3710-4293-ba2b-73982bcf0e84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n"
Sep 09 20:32:36.854033 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.853432    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-host-var-lib-cni-bin\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.854033 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.853435    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-host-var-lib-kubelet\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.854033 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.853449    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/86f6ae47-3710-4293-ba2b-73982bcf0e84-socket-dir\") pod \"aws-ebs-csi-driver-node-rmn7n\" (UID: \"86f6ae47-3710-4293-ba2b-73982bcf0e84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n"
Sep 09 20:32:36.854033 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.853464    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/61bb5c63-cbb7-4186-9c23-5706dba8cec2-dbus\") pod \"global-pull-secret-syncer-q2zw5\" (UID: \"61bb5c63-cbb7-4186-9c23-5706dba8cec2\") " pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:36.854033 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.853488    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-host-var-lib-cni-multus\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.854033 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.853494    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/61416974-f481-4e36-ab4a-2be050c06afe-hosts-file\") pod \"node-resolver-bmnnt\" (UID: \"61416974-f481-4e36-ab4a-2be050c06afe\") " pod="openshift-dns/node-resolver-bmnnt"
Sep 09 20:32:36.854033 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.853498    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/61bb5c63-cbb7-4186-9c23-5706dba8cec2-kubelet-config\") pod \"global-pull-secret-syncer-q2zw5\" (UID: \"61bb5c63-cbb7-4186-9c23-5706dba8cec2\") " pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:36.854815 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.853534    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21bf48f4-01d2-4e62-af48-52956be58f15-cnibin\") pod \"multus-additional-cni-plugins-m86gm\" (UID: \"21bf48f4-01d2-4e62-af48-52956be58f15\") " pod="openshift-multus/multus-additional-cni-plugins-m86gm"
Sep 09 20:32:36.854815 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.853582    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-system-cni-dir\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.854815 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.853804    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-multus-daemon-config\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.854815 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.854365    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65da59e2-9681-4904-bd9a-ed612f4d54b9-iptables-alerter-script\") pod \"iptables-alerter-9rvmd\" (UID: \"65da59e2-9681-4904-bd9a-ed612f4d54b9\") " pod="openshift-network-operator/iptables-alerter-9rvmd"
Sep 09 20:32:36.860135 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.860112    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9khkh\" (UniqueName: \"kubernetes.io/projected/c31d1698-1b9c-451e-995a-ed666693b6a2-kube-api-access-9khkh\") pod \"node-ca-kj4jd\" (UID: \"c31d1698-1b9c-451e-995a-ed666693b6a2\") " pod="openshift-image-registry/node-ca-kj4jd"
Sep 09 20:32:36.862530 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.862476    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dphwt\" (UniqueName: \"kubernetes.io/projected/8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3-kube-api-access-dphwt\") pod \"multus-wkmrq\" (UID: \"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3\") " pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.863269 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.862712    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc6xq\" (UniqueName: \"kubernetes.io/projected/21bf48f4-01d2-4e62-af48-52956be58f15-kube-api-access-jc6xq\") pod \"multus-additional-cni-plugins-m86gm\" (UID: \"21bf48f4-01d2-4e62-af48-52956be58f15\") " pod="openshift-multus/multus-additional-cni-plugins-m86gm"
Sep 09 20:32:36.863269 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.863221    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2ntl\" (UniqueName: \"kubernetes.io/projected/86f6ae47-3710-4293-ba2b-73982bcf0e84-kube-api-access-d2ntl\") pod \"aws-ebs-csi-driver-node-rmn7n\" (UID: \"86f6ae47-3710-4293-ba2b-73982bcf0e84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n"
Sep 09 20:32:36.864004 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.863978    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhqkt\" (UniqueName: \"kubernetes.io/projected/61416974-f481-4e36-ab4a-2be050c06afe-kube-api-access-fhqkt\") pod \"node-resolver-bmnnt\" (UID: \"61416974-f481-4e36-ab4a-2be050c06afe\") " pod="openshift-dns/node-resolver-bmnnt"
Sep 09 20:32:36.864618 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.864578    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5lpw\" (UniqueName: \"kubernetes.io/projected/65da59e2-9681-4904-bd9a-ed612f4d54b9-kube-api-access-j5lpw\") pod \"iptables-alerter-9rvmd\" (UID: \"65da59e2-9681-4904-bd9a-ed612f4d54b9\") " pod="openshift-network-operator/iptables-alerter-9rvmd"
Sep 09 20:32:36.891847 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.891803    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal" event={"ID":"3508d0a07aa0fd49c57a7f8e3fd46334","Type":"ContainerStarted","Data":"36412708256f6f81261cfd3112d323cf6033b62d1fbe6ade506630241418d46a"}
Sep 09 20:32:36.893591 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.893568    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-117.ec2.internal" event={"ID":"4a08d7fc3d09b5f1580b5cd035f70a19","Type":"ContainerStarted","Data":"0ed22847f5080b9cc2d10b4f66bd84622001b04aca01b1e1e4a4ad0bc785d3a9"}
Sep 09 20:32:36.936981 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.936927    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w8v2w"
Sep 09 20:32:36.947181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.947149    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jr87x"
Sep 09 20:32:36.949455 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:36.949424    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8c3dbb_49b9_43e4_94d0_0beaa80b3e40.slice/crio-8c28771d7e7563443c7a2c3e2deb9837a1f004d515c4c5710a12cf621011e3fd WatchSource:0}: Error finding container 8c28771d7e7563443c7a2c3e2deb9837a1f004d515c4c5710a12cf621011e3fd: Status 404 returned error can't find the container with id 8c28771d7e7563443c7a2c3e2deb9837a1f004d515c4c5710a12cf621011e3fd
Sep 09 20:32:36.956412 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.956353    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:32:36.962246 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.962221    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kj4jd"
Sep 09 20:32:36.964236 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:36.964211    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7241694e_5bba_4400_9eb9_771af5989280.slice/crio-47005b2a9bc41b8aa5bf8dba15c056ab51db007f179831cdea88877f4f39ff1b WatchSource:0}: Error finding container 47005b2a9bc41b8aa5bf8dba15c056ab51db007f179831cdea88877f4f39ff1b: Status 404 returned error can't find the container with id 47005b2a9bc41b8aa5bf8dba15c056ab51db007f179831cdea88877f4f39ff1b
Sep 09 20:32:36.968418 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.968397    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bmnnt"
Sep 09 20:32:36.975886 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.975867    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n"
Sep 09 20:32:36.983195 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.982236    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m86gm"
Sep 09 20:32:36.984198 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:36.984176    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31d1698_1b9c_451e_995a_ed666693b6a2.slice/crio-b1ea4e43d3b9fc58afd1af41710456e0c6fb4a19d2a97f53f69fce427e7f508d WatchSource:0}: Error finding container b1ea4e43d3b9fc58afd1af41710456e0c6fb4a19d2a97f53f69fce427e7f508d: Status 404 returned error can't find the container with id b1ea4e43d3b9fc58afd1af41710456e0c6fb4a19d2a97f53f69fce427e7f508d
Sep 09 20:32:36.984748 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:36.984722    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61416974_f481_4e36_ab4a_2be050c06afe.slice/crio-c3bc65fe1b3d74fc23555cec40c48d6b228a60fe74799af50410a1af5e0d5173 WatchSource:0}: Error finding container c3bc65fe1b3d74fc23555cec40c48d6b228a60fe74799af50410a1af5e0d5173: Status 404 returned error can't find the container with id c3bc65fe1b3d74fc23555cec40c48d6b228a60fe74799af50410a1af5e0d5173
Sep 09 20:32:36.989526 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:36.988998    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f6ae47_3710_4293_ba2b_73982bcf0e84.slice/crio-3451165098d3920c0ac1e13fce05886f27bfef4527abcca71099ab823ba6b11d WatchSource:0}: Error finding container 3451165098d3920c0ac1e13fce05886f27bfef4527abcca71099ab823ba6b11d: Status 404 returned error can't find the container with id 3451165098d3920c0ac1e13fce05886f27bfef4527abcca71099ab823ba6b11d
Sep 09 20:32:36.989766 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.989748    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wkmrq"
Sep 09 20:32:36.994580 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:36.994393    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21bf48f4_01d2_4e62_af48_52956be58f15.slice/crio-86c01cd727757aa368099941d0a764384e76099657bc524cdef2d6c97aef8875 WatchSource:0}: Error finding container 86c01cd727757aa368099941d0a764384e76099657bc524cdef2d6c97aef8875: Status 404 returned error can't find the container with id 86c01cd727757aa368099941d0a764384e76099657bc524cdef2d6c97aef8875
Sep 09 20:32:36.996626 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:36.996479    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9rvmd"
Sep 09 20:32:37.001382 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:37.001354    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1a2e29_fbc8_4b78_99d0_ae382f56d9f3.slice/crio-7c4e72af06cf8c14b15170e96f57b2d1449e67204001f3497db9d21af7e3b618 WatchSource:0}: Error finding container 7c4e72af06cf8c14b15170e96f57b2d1449e67204001f3497db9d21af7e3b618: Status 404 returned error can't find the container with id 7c4e72af06cf8c14b15170e96f57b2d1449e67204001f3497db9d21af7e3b618
Sep 09 20:32:37.007816 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:32:37.007760    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65da59e2_9681_4904_bd9a_ed612f4d54b9.slice/crio-538ee3c1803f9389d0754ae637491868f0bb2c7c1b2c8e0b8f28b730044bfc5a WatchSource:0}: Error finding container 538ee3c1803f9389d0754ae637491868f0bb2c7c1b2c8e0b8f28b730044bfc5a: Status 404 returned error can't find the container with id 538ee3c1803f9389d0754ae637491868f0bb2c7c1b2c8e0b8f28b730044bfc5a
Sep 09 20:32:37.254833 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:37.254789    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs\") pod \"network-metrics-daemon-bwnpx\" (UID: \"73ad4e42-0fc1-4d62-afcf-596c76c4ea87\") " pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:32:37.255015 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:37.254963    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:32:37.255080 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:37.255043    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs podName:73ad4e42-0fc1-4d62-afcf-596c76c4ea87 nodeName:}" failed. No retries permitted until 2026-09-09 20:32:38.255022078 +0000 UTC m=+3.934982367 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs") pod "network-metrics-daemon-bwnpx" (UID: "73ad4e42-0fc1-4d62-afcf-596c76c4ea87") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:32:37.355301 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:37.355273    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj47h\" (UniqueName: \"kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h\") pod \"network-check-target-2zpmx\" (UID: \"14c3f4cb-c792-4945-bd65-932ed6a27aa9\") " pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:32:37.355473 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:37.355350    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret\") pod \"global-pull-secret-syncer-q2zw5\" (UID: \"61bb5c63-cbb7-4186-9c23-5706dba8cec2\") " pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:37.355473 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:37.355439    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:32:37.355473 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:37.355459    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:32:37.355632 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:37.355518    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret podName:61bb5c63-cbb7-4186-9c23-5706dba8cec2 nodeName:}" failed. No retries permitted until 2026-09-09 20:32:38.355500258 +0000 UTC m=+4.035460562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret") pod "global-pull-secret-syncer-q2zw5" (UID: "61bb5c63-cbb7-4186-9c23-5706dba8cec2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:32:37.355632 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:37.355463    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:32:37.355632 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:37.355546    2556 projected.go:196] Error preparing data for projected volume kube-api-access-nj47h for pod openshift-network-diagnostics/network-check-target-2zpmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:32:37.355632 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:37.355598    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h podName:14c3f4cb-c792-4945-bd65-932ed6a27aa9 nodeName:}" failed. No retries permitted until 2026-09-09 20:32:38.355583876 +0000 UTC m=+4.035544165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nj47h" (UniqueName: "kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h") pod "network-check-target-2zpmx" (UID: "14c3f4cb-c792-4945-bd65-932ed6a27aa9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:32:37.730660 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:37.730551    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:27:36 +0000 UTC" deadline="2028-05-22 13:30:35.529541754 +0000 UTC"
Sep 09 20:32:37.730660 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:37.730612    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14896h57m57.798933227s"
Sep 09 20:32:37.886686 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:37.886657    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:32:37.886825 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:37.886799    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zpmx" podUID="14c3f4cb-c792-4945-bd65-932ed6a27aa9"
Sep 09 20:32:37.897217 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:37.897183    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bmnnt" event={"ID":"61416974-f481-4e36-ab4a-2be050c06afe","Type":"ContainerStarted","Data":"c3bc65fe1b3d74fc23555cec40c48d6b228a60fe74799af50410a1af5e0d5173"}
Sep 09 20:32:37.898600 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:37.898551    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kj4jd" event={"ID":"c31d1698-1b9c-451e-995a-ed666693b6a2","Type":"ContainerStarted","Data":"b1ea4e43d3b9fc58afd1af41710456e0c6fb4a19d2a97f53f69fce427e7f508d"}
Sep 09 20:32:37.899925 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:37.899792    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w8v2w" event={"ID":"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40","Type":"ContainerStarted","Data":"8c28771d7e7563443c7a2c3e2deb9837a1f004d515c4c5710a12cf621011e3fd"}
Sep 09 20:32:37.901076 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:37.901049    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9rvmd" event={"ID":"65da59e2-9681-4904-bd9a-ed612f4d54b9","Type":"ContainerStarted","Data":"538ee3c1803f9389d0754ae637491868f0bb2c7c1b2c8e0b8f28b730044bfc5a"}
Sep 09 20:32:37.902628 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:37.902606    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m86gm" event={"ID":"21bf48f4-01d2-4e62-af48-52956be58f15","Type":"ContainerStarted","Data":"86c01cd727757aa368099941d0a764384e76099657bc524cdef2d6c97aef8875"}
Sep 09 20:32:37.904406 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:37.904381    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-25h4s" event={"ID":"33ba58cf-b5d2-4b22-83d2-ef5e54576651","Type":"ContainerStarted","Data":"3aaa3847c88a23631247533b7ebbb557c603c213460cd77bc395ecadc4cad38c"}
Sep 09 20:32:37.906088 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:37.906062    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jr87x" event={"ID":"7241694e-5bba-4400-9eb9-771af5989280","Type":"ContainerStarted","Data":"47005b2a9bc41b8aa5bf8dba15c056ab51db007f179831cdea88877f4f39ff1b"}
Sep 09 20:32:37.907416 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:37.907394    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wkmrq" event={"ID":"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3","Type":"ContainerStarted","Data":"7c4e72af06cf8c14b15170e96f57b2d1449e67204001f3497db9d21af7e3b618"}
Sep 09 20:32:37.908620 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:37.908597    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n" event={"ID":"86f6ae47-3710-4293-ba2b-73982bcf0e84","Type":"ContainerStarted","Data":"3451165098d3920c0ac1e13fce05886f27bfef4527abcca71099ab823ba6b11d"}
Sep 09 20:32:38.263122 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:38.262566    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs\") pod \"network-metrics-daemon-bwnpx\" (UID: \"73ad4e42-0fc1-4d62-afcf-596c76c4ea87\") " pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:32:38.263122 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:38.262712    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:32:38.263122 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:38.262773    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs podName:73ad4e42-0fc1-4d62-afcf-596c76c4ea87 nodeName:}" failed. No retries permitted until 2026-09-09 20:32:40.26275546 +0000 UTC m=+5.942715759 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs") pod "network-metrics-daemon-bwnpx" (UID: "73ad4e42-0fc1-4d62-afcf-596c76c4ea87") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:32:38.364282 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:38.363469    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret\") pod \"global-pull-secret-syncer-q2zw5\" (UID: \"61bb5c63-cbb7-4186-9c23-5706dba8cec2\") " pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:38.364282 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:38.363531    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj47h\" (UniqueName: \"kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h\") pod \"network-check-target-2zpmx\" (UID: \"14c3f4cb-c792-4945-bd65-932ed6a27aa9\") " pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:32:38.364282 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:38.363681    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:32:38.364282 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:38.363703    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:32:38.364282 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:38.363716    2556 projected.go:196] Error preparing data for projected volume kube-api-access-nj47h for pod openshift-network-diagnostics/network-check-target-2zpmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:32:38.364282 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:38.363770    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h podName:14c3f4cb-c792-4945-bd65-932ed6a27aa9 nodeName:}" failed. No retries permitted until 2026-09-09 20:32:40.36375355 +0000 UTC m=+6.043713856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nj47h" (UniqueName: "kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h") pod "network-check-target-2zpmx" (UID: "14c3f4cb-c792-4945-bd65-932ed6a27aa9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:32:38.364282 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:38.364198    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:32:38.364282 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:38.364250    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret podName:61bb5c63-cbb7-4186-9c23-5706dba8cec2 nodeName:}" failed. No retries permitted until 2026-09-09 20:32:40.364234378 +0000 UTC m=+6.044194667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret") pod "global-pull-secret-syncer-q2zw5" (UID: "61bb5c63-cbb7-4186-9c23-5706dba8cec2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:32:38.722763 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:38.721688    2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:32:38.731704 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:38.731586    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:27:36 +0000 UTC" deadline="2028-05-14 20:21:31.96516073 +0000 UTC"
Sep 09 20:32:38.731704 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:38.731619    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14711h48m53.233547748s"
Sep 09 20:32:38.888106 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:38.888073    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:38.888106 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:38.888098    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:32:38.888271 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:38.888238    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zw5" podUID="61bb5c63-cbb7-4186-9c23-5706dba8cec2"
Sep 09 20:32:38.888744 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:38.888718    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwnpx" podUID="73ad4e42-0fc1-4d62-afcf-596c76c4ea87"
Sep 09 20:32:39.886685 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:39.886145    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:32:39.886685 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:39.886282    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zpmx" podUID="14c3f4cb-c792-4945-bd65-932ed6a27aa9"
Sep 09 20:32:39.947288 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:39.947253    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal" event={"ID":"3508d0a07aa0fd49c57a7f8e3fd46334","Type":"ContainerStarted","Data":"19c1bf8d79ab24f33011672e1764af365f07dc0e5098d3aed572af0886e86560"}
Sep 09 20:32:39.961196 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:39.960745    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-117.ec2.internal" event={"ID":"4a08d7fc3d09b5f1580b5cd035f70a19","Type":"ContainerStarted","Data":"b379292d13d1f318885f8addf7b307bf7450cd710c0e3208de908781aa143a1d"}
Sep 09 20:32:40.281303 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:40.281264    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs\") pod \"network-metrics-daemon-bwnpx\" (UID: \"73ad4e42-0fc1-4d62-afcf-596c76c4ea87\") " pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:32:40.281490 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:40.281430    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:32:40.281490 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:40.281488    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs podName:73ad4e42-0fc1-4d62-afcf-596c76c4ea87 nodeName:}" failed. No retries permitted until 2026-09-09 20:32:44.281471602 +0000 UTC m=+9.961431903 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs") pod "network-metrics-daemon-bwnpx" (UID: "73ad4e42-0fc1-4d62-afcf-596c76c4ea87") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:32:40.381647 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:40.381611    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj47h\" (UniqueName: \"kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h\") pod \"network-check-target-2zpmx\" (UID: \"14c3f4cb-c792-4945-bd65-932ed6a27aa9\") " pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:32:40.381813 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:40.381684    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret\") pod \"global-pull-secret-syncer-q2zw5\" (UID: \"61bb5c63-cbb7-4186-9c23-5706dba8cec2\") " pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:40.381879 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:40.381812    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:32:40.381879 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:40.381873    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret podName:61bb5c63-cbb7-4186-9c23-5706dba8cec2 nodeName:}" failed. No retries permitted until 2026-09-09 20:32:44.381852942 +0000 UTC m=+10.061813231 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret") pod "global-pull-secret-syncer-q2zw5" (UID: "61bb5c63-cbb7-4186-9c23-5706dba8cec2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:32:40.382301 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:40.382279    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:32:40.382404 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:40.382305    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:32:40.382404 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:40.382317    2556 projected.go:196] Error preparing data for projected volume kube-api-access-nj47h for pod openshift-network-diagnostics/network-check-target-2zpmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:32:40.382404 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:40.382360    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h podName:14c3f4cb-c792-4945-bd65-932ed6a27aa9 nodeName:}" failed. No retries permitted until 2026-09-09 20:32:44.382345753 +0000 UTC m=+10.062306046 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nj47h" (UniqueName: "kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h") pod "network-check-target-2zpmx" (UID: "14c3f4cb-c792-4945-bd65-932ed6a27aa9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:32:40.887043 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:40.887013    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:32:40.887043 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:40.887045    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:40.887655 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:40.887149    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwnpx" podUID="73ad4e42-0fc1-4d62-afcf-596c76c4ea87"
Sep 09 20:32:40.887655 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:40.887631    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zw5" podUID="61bb5c63-cbb7-4186-9c23-5706dba8cec2"
Sep 09 20:32:41.886162 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:41.886134    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:32:41.886327 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:41.886256    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zpmx" podUID="14c3f4cb-c792-4945-bd65-932ed6a27aa9"
Sep 09 20:32:42.887036 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:42.886994    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:32:42.887491 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:42.887102    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwnpx" podUID="73ad4e42-0fc1-4d62-afcf-596c76c4ea87"
Sep 09 20:32:42.887491 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:42.887363    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:42.887491 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:42.887439    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zw5" podUID="61bb5c63-cbb7-4186-9c23-5706dba8cec2"
Sep 09 20:32:43.886392 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:43.886354    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:32:43.886564 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:43.886467    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zpmx" podUID="14c3f4cb-c792-4945-bd65-932ed6a27aa9"
Sep 09 20:32:43.970987 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:43.970946    2556 generic.go:356] "Generic (PLEG): container finished" podID="3508d0a07aa0fd49c57a7f8e3fd46334" containerID="19c1bf8d79ab24f33011672e1764af365f07dc0e5098d3aed572af0886e86560" exitCode=0
Sep 09 20:32:43.971493 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:43.970996    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal" event={"ID":"3508d0a07aa0fd49c57a7f8e3fd46334","Type":"ContainerDied","Data":"19c1bf8d79ab24f33011672e1764af365f07dc0e5098d3aed572af0886e86560"}
Sep 09 20:32:43.987242 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:43.987125    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-117.ec2.internal" podStartSLOduration=8.987105153 podStartE2EDuration="8.987105153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:32:39.981784391 +0000 UTC m=+5.661744702" watchObservedRunningTime="2026-09-09 20:32:43.987105153 +0000 UTC m=+9.667065462"
Sep 09 20:32:44.317503 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:44.317372    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs\") pod \"network-metrics-daemon-bwnpx\" (UID: \"73ad4e42-0fc1-4d62-afcf-596c76c4ea87\") " pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:32:44.317682 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:44.317541    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:32:44.317682 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:44.317618    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs podName:73ad4e42-0fc1-4d62-afcf-596c76c4ea87 nodeName:}" failed. No retries permitted until 2026-09-09 20:32:52.317597895 +0000 UTC m=+17.997558189 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs") pod "network-metrics-daemon-bwnpx" (UID: "73ad4e42-0fc1-4d62-afcf-596c76c4ea87") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:32:44.418367 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:44.418329    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret\") pod \"global-pull-secret-syncer-q2zw5\" (UID: \"61bb5c63-cbb7-4186-9c23-5706dba8cec2\") " pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:44.418536 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:44.418392    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj47h\" (UniqueName: \"kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h\") pod \"network-check-target-2zpmx\" (UID: \"14c3f4cb-c792-4945-bd65-932ed6a27aa9\") " pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:32:44.418536 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:44.418507    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:32:44.418536 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:44.418530    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:32:44.418706 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:44.418550    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:32:44.418706 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:44.418561    2556 projected.go:196] Error preparing data for projected volume kube-api-access-nj47h for pod openshift-network-diagnostics/network-check-target-2zpmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:32:44.418706 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:44.418587    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret podName:61bb5c63-cbb7-4186-9c23-5706dba8cec2 nodeName:}" failed. No retries permitted until 2026-09-09 20:32:52.418568162 +0000 UTC m=+18.098528464 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret") pod "global-pull-secret-syncer-q2zw5" (UID: "61bb5c63-cbb7-4186-9c23-5706dba8cec2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:32:44.418706 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:44.418608    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h podName:14c3f4cb-c792-4945-bd65-932ed6a27aa9 nodeName:}" failed. No retries permitted until 2026-09-09 20:32:52.418598698 +0000 UTC m=+18.098558998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nj47h" (UniqueName: "kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h") pod "network-check-target-2zpmx" (UID: "14c3f4cb-c792-4945-bd65-932ed6a27aa9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:32:44.887231 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:44.887197    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:32:44.887423 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:44.887322    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwnpx" podUID="73ad4e42-0fc1-4d62-afcf-596c76c4ea87"
Sep 09 20:32:44.887423 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:44.887360    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:44.887539 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:44.887427    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zw5" podUID="61bb5c63-cbb7-4186-9c23-5706dba8cec2"
Sep 09 20:32:45.886662 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:45.886548    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:32:45.887167 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:45.886666    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zpmx" podUID="14c3f4cb-c792-4945-bd65-932ed6a27aa9"
Sep 09 20:32:46.886673 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:46.886629    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:46.887144 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:46.886761    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zw5" podUID="61bb5c63-cbb7-4186-9c23-5706dba8cec2"
Sep 09 20:32:46.887144 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:46.886852    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:32:46.887144 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:46.886985    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwnpx" podUID="73ad4e42-0fc1-4d62-afcf-596c76c4ea87"
Sep 09 20:32:47.886354 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:47.886281    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:32:47.886508 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:47.886390    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zpmx" podUID="14c3f4cb-c792-4945-bd65-932ed6a27aa9"
Sep 09 20:32:48.886238 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:48.886202    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:48.886649 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:48.886340    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zw5" podUID="61bb5c63-cbb7-4186-9c23-5706dba8cec2"
Sep 09 20:32:48.886649 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:48.886403    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:32:48.886649 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:48.886539    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwnpx" podUID="73ad4e42-0fc1-4d62-afcf-596c76c4ea87"
Sep 09 20:32:49.886094 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:49.886053    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:32:49.886264 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:49.886182    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zpmx" podUID="14c3f4cb-c792-4945-bd65-932ed6a27aa9"
Sep 09 20:32:50.886299 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:50.886260    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:50.886299 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:50.886277    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:32:50.886775 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:50.886374    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zw5" podUID="61bb5c63-cbb7-4186-9c23-5706dba8cec2"
Sep 09 20:32:50.886775 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:50.886523    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwnpx" podUID="73ad4e42-0fc1-4d62-afcf-596c76c4ea87"
Sep 09 20:32:51.886214 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:51.886172    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:32:51.886416 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:51.886305    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zpmx" podUID="14c3f4cb-c792-4945-bd65-932ed6a27aa9"
Sep 09 20:32:52.376495 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:52.376458    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs\") pod \"network-metrics-daemon-bwnpx\" (UID: \"73ad4e42-0fc1-4d62-afcf-596c76c4ea87\") " pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:32:52.376777 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:52.376622    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:32:52.376777 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:52.376691    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs podName:73ad4e42-0fc1-4d62-afcf-596c76c4ea87 nodeName:}" failed. No retries permitted until 2026-09-09 20:33:08.376674748 +0000 UTC m=+34.056635038 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs") pod "network-metrics-daemon-bwnpx" (UID: "73ad4e42-0fc1-4d62-afcf-596c76c4ea87") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:32:52.477802 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:52.477766    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret\") pod \"global-pull-secret-syncer-q2zw5\" (UID: \"61bb5c63-cbb7-4186-9c23-5706dba8cec2\") " pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:52.477984 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:52.477830    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj47h\" (UniqueName: \"kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h\") pod \"network-check-target-2zpmx\" (UID: \"14c3f4cb-c792-4945-bd65-932ed6a27aa9\") " pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:32:52.477984 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:52.477946    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:32:52.478084 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:52.477986    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:32:52.478084 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:52.478003    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:32:52.478084 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:52.478014    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret podName:61bb5c63-cbb7-4186-9c23-5706dba8cec2 nodeName:}" failed. No retries permitted until 2026-09-09 20:33:08.477998019 +0000 UTC m=+34.157958310 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret") pod "global-pull-secret-syncer-q2zw5" (UID: "61bb5c63-cbb7-4186-9c23-5706dba8cec2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:32:52.478084 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:52.478016    2556 projected.go:196] Error preparing data for projected volume kube-api-access-nj47h for pod openshift-network-diagnostics/network-check-target-2zpmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:32:52.478084 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:52.478069    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h podName:14c3f4cb-c792-4945-bd65-932ed6a27aa9 nodeName:}" failed. No retries permitted until 2026-09-09 20:33:08.478053604 +0000 UTC m=+34.158013905 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nj47h" (UniqueName: "kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h") pod "network-check-target-2zpmx" (UID: "14c3f4cb-c792-4945-bd65-932ed6a27aa9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:32:52.886108 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:52.886072    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:52.886290 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:52.886213    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zw5" podUID="61bb5c63-cbb7-4186-9c23-5706dba8cec2"
Sep 09 20:32:52.886290 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:52.886266    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:32:52.886405 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:52.886354    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwnpx" podUID="73ad4e42-0fc1-4d62-afcf-596c76c4ea87"
Sep 09 20:32:53.886455 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:53.886424    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:32:53.886849 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:53.886524    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zpmx" podUID="14c3f4cb-c792-4945-bd65-932ed6a27aa9"
Sep 09 20:32:54.887107 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:54.887077    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:32:54.887432 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:54.887171    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwnpx" podUID="73ad4e42-0fc1-4d62-afcf-596c76c4ea87"
Sep 09 20:32:54.887432 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:54.887262    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:54.887432 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:54.887346    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zw5" podUID="61bb5c63-cbb7-4186-9c23-5706dba8cec2"
Sep 09 20:32:55.886896 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:55.886530    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:32:55.887380 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:55.887066    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zpmx" podUID="14c3f4cb-c792-4945-bd65-932ed6a27aa9"
Sep 09 20:32:55.991863 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:55.991833    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w8v2w" event={"ID":"ee8c3dbb-49b9-43e4-94d0-0beaa80b3e40","Type":"ContainerStarted","Data":"274a4f0ae0a2e88502e2705bd6d47b034df3a28587ce2c244228d3731bfdf4cb"}
Sep 09 20:32:55.993939 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:55.993898    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal" event={"ID":"3508d0a07aa0fd49c57a7f8e3fd46334","Type":"ContainerStarted","Data":"2e5aa9d72aecf3596f7d4035a8003c71106e1e5f859b1aaf25a56ea8d56f944c"}
Sep 09 20:32:55.995520 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:55.995492    2556 generic.go:356] "Generic (PLEG): container finished" podID="21bf48f4-01d2-4e62-af48-52956be58f15" containerID="ca6f1fbf600d3ce149f9f18593c2783e5d0cc0de0e8748b14c5cc160692e17b9" exitCode=0
Sep 09 20:32:55.995682 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:55.995577    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m86gm" event={"ID":"21bf48f4-01d2-4e62-af48-52956be58f15","Type":"ContainerDied","Data":"ca6f1fbf600d3ce149f9f18593c2783e5d0cc0de0e8748b14c5cc160692e17b9"}
Sep 09 20:32:55.999141 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:55.999121    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-25h4s" event={"ID":"33ba58cf-b5d2-4b22-83d2-ef5e54576651","Type":"ContainerStarted","Data":"572abe4a06743451b069fa2501269fd60969dcaeaf5f170c70c17a73a66ecfd1"}
Sep 09 20:32:55.999243 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:55.999192    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-25h4s" event={"ID":"33ba58cf-b5d2-4b22-83d2-ef5e54576651","Type":"ContainerStarted","Data":"855f6650dc5955d56c69773a1dea96ac4bbe0359557d3b86fb6c2ec353abe814"}
Sep 09 20:32:55.999243 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:55.999231    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-25h4s" event={"ID":"33ba58cf-b5d2-4b22-83d2-ef5e54576651","Type":"ContainerStarted","Data":"93c73b6b2b093e5c83a392b4342d03a68e556968c80af92d1952943e996ae5ef"}
Sep 09 20:32:55.999346 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:55.999264    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-25h4s" event={"ID":"33ba58cf-b5d2-4b22-83d2-ef5e54576651","Type":"ContainerStarted","Data":"c369d5e50acbaf7b5ed20bdb73a9a5ba89b07850c9dc0d823fd6a505bae54a2a"}
Sep 09 20:32:55.999346 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:55.999292    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-25h4s" event={"ID":"33ba58cf-b5d2-4b22-83d2-ef5e54576651","Type":"ContainerStarted","Data":"1b93a2ad5458587a85c62b8cd573905f9e924cde9296699adfb81bb2b55206be"}
Sep 09 20:32:55.999346 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:55.999319    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-25h4s" event={"ID":"33ba58cf-b5d2-4b22-83d2-ef5e54576651","Type":"ContainerStarted","Data":"a1b9080b729a76c7177768027467740a15348267310351bc9fdf2ea50d9082fb"}
Sep 09 20:32:56.000721 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:56.000685    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jr87x" event={"ID":"7241694e-5bba-4400-9eb9-771af5989280","Type":"ContainerStarted","Data":"03837fb670a64846775caa599bb8d2c9074ac085742b2fa6abc790802164cbef"}
Sep 09 20:32:56.002308 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:56.002272    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wkmrq" event={"ID":"8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3","Type":"ContainerStarted","Data":"20eeead2c501d06695e64aacf7575296bd8fc19c482a29e515dee70133996741"}
Sep 09 20:32:56.003853 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:56.003833    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n" event={"ID":"86f6ae47-3710-4293-ba2b-73982bcf0e84","Type":"ContainerStarted","Data":"0fb8d5dc2b108c762cfa592322749ce44c74893ddc74c93a9731d44b3afde0f3"}
Sep 09 20:32:56.005460 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:56.005437    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bmnnt" event={"ID":"61416974-f481-4e36-ab4a-2be050c06afe","Type":"ContainerStarted","Data":"f9a7c71a0af070ac2c84b0f0e7dfe6fafff4182af883caf841ff062ed7c0051a"}
Sep 09 20:32:56.006826 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:56.006810    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kj4jd" event={"ID":"c31d1698-1b9c-451e-995a-ed666693b6a2","Type":"ContainerStarted","Data":"3dcdc10b11602410949d2deaba9df5ba2937b6bf4800a72670a3256c56c79b3d"}
Sep 09 20:32:56.009740 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:56.009702    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-w8v2w" podStartSLOduration=3.792664323 podStartE2EDuration="22.009689526s" totalImagesPullingTime="18.217025203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:32:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:32:56.009314293 +0000 UTC m=+21.689274602" watchObservedRunningTime="2026-09-09 20:32:56.009689526 +0000 UTC m=+21.689649834"
Sep 09 20:32:56.023984 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:56.023943    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jr87x" podStartSLOduration=11.322940542 podStartE2EDuration="22.02392831s" totalImagesPullingTime="10.700987768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:32:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:32:56.023843293 +0000 UTC m=+21.703803603" watchObservedRunningTime="2026-09-09 20:32:56.02392831 +0000 UTC m=+21.703888623"
Sep 09 20:32:56.057325 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:56.057286    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kj4jd" podStartSLOduration=3.891914263 podStartE2EDuration="22.057274497s" totalImagesPullingTime="18.165360234s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:32:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:32:56.056915665 +0000 UTC m=+21.736875967" watchObservedRunningTime="2026-09-09 20:32:56.057274497 +0000 UTC m=+21.737234805"
Sep 09 20:32:56.070700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:56.070665    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bmnnt" podStartSLOduration=2.905733175 podStartE2EDuration="21.070656235s" totalImagesPullingTime="18.16492306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:32:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:32:56.070068639 +0000 UTC m=+21.750028950" watchObservedRunningTime="2026-09-09 20:32:56.070656235 +0000 UTC m=+21.750616545"
Sep 09 20:32:56.109564 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:56.109526    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal" podStartSLOduration=21.109514847 podStartE2EDuration="21.109514847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:32:56.10922474 +0000 UTC m=+21.789185050" watchObservedRunningTime="2026-09-09 20:32:56.109514847 +0000 UTC m=+21.789475155"
Sep 09 20:32:56.109702 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:56.109675    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wkmrq" podStartSLOduration=2.891177715 podStartE2EDuration="21.109666924s" totalImagesPullingTime="18.218489209s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:32:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:32:56.095357987 +0000 UTC m=+21.775318296" watchObservedRunningTime="2026-09-09 20:32:56.109666924 +0000 UTC m=+21.789627235"
Sep 09 20:32:56.312366 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:56.312343    2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 20:32:56.843650 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:56.843321    2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:32:56.312362867Z","UUID":"b098ecb5-e419-44d3-b001-9d7a0469151e","Handler":null,"Name":"","Endpoint":""}
Sep 09 20:32:56.845168 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:56.845060    2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 20:32:56.845168 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:56.845093    2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 20:32:56.886976 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:56.886951    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:56.887102 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:56.887076    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zw5" podUID="61bb5c63-cbb7-4186-9c23-5706dba8cec2"
Sep 09 20:32:56.887102 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:56.887093    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:32:56.887229 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:56.887209    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwnpx" podUID="73ad4e42-0fc1-4d62-afcf-596c76c4ea87"
Sep 09 20:32:57.010628 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:57.010588    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n" event={"ID":"86f6ae47-3710-4293-ba2b-73982bcf0e84","Type":"ContainerStarted","Data":"c1a879c01a1210eedf5671938b6f9562cf904a08f8104ff33917da97908bb1e0"}
Sep 09 20:32:57.012401 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:57.012344    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9rvmd" event={"ID":"65da59e2-9681-4904-bd9a-ed612f4d54b9","Type":"ContainerStarted","Data":"f9e05c2cc94f76d8b61e39cfc32baa8f2bce35f12e32cc42c542db7d0069e892"}
Sep 09 20:32:57.028184 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:57.028141    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9rvmd" podStartSLOduration=3.866361591 podStartE2EDuration="22.028119817s" totalImagesPullingTime="18.161758226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:32:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:32:57.027842134 +0000 UTC m=+22.707802444" watchObservedRunningTime="2026-09-09 20:32:57.028119817 +0000 UTC m=+22.708080117"
Sep 09 20:32:57.886069 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:57.886038    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:32:57.886253 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:57.886157    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zpmx" podUID="14c3f4cb-c792-4945-bd65-932ed6a27aa9"
Sep 09 20:32:58.019446 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:58.019420    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-25h4s" event={"ID":"33ba58cf-b5d2-4b22-83d2-ef5e54576651","Type":"ContainerStarted","Data":"59fd161853d7aaa321113062a49ec80309fcf49c439f47ea6e3fa6e24837e90c"}
Sep 09 20:32:58.021885 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:58.021857    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n" event={"ID":"86f6ae47-3710-4293-ba2b-73982bcf0e84","Type":"ContainerStarted","Data":"45931a23b3ac097d09052d0a719ae377828ee0aabc2622180582c28835197353"}
Sep 09 20:32:58.040315 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:58.040267    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmn7n" podStartSLOduration=3.114019312 podStartE2EDuration="23.040255134s" totalImagesPullingTime="19.926235822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:32:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:32:58.040184134 +0000 UTC m=+23.720144443" watchObservedRunningTime="2026-09-09 20:32:58.040255134 +0000 UTC m=+23.720215484"
Sep 09 20:32:58.887020 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:58.886990    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:32:58.887223 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:58.887087    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zw5" podUID="61bb5c63-cbb7-4186-9c23-5706dba8cec2"
Sep 09 20:32:58.887223 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:58.887192    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:32:58.887334 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:58.887312    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwnpx" podUID="73ad4e42-0fc1-4d62-afcf-596c76c4ea87"
Sep 09 20:32:59.886348 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:32:59.886321    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:32:59.886823 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:32:59.886425    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zpmx" podUID="14c3f4cb-c792-4945-bd65-932ed6a27aa9"
Sep 09 20:33:00.886705 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:00.886573    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:33:00.887150 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:00.886634    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:33:00.887150 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:00.886789    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwnpx" podUID="73ad4e42-0fc1-4d62-afcf-596c76c4ea87"
Sep 09 20:33:00.887150 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:00.886852    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zw5" podUID="61bb5c63-cbb7-4186-9c23-5706dba8cec2"
Sep 09 20:33:00.891404 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:00.891386    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jr87x"
Sep 09 20:33:00.891843 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:00.891827    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jr87x"
Sep 09 20:33:01.035284 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:01.035230    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-25h4s" event={"ID":"33ba58cf-b5d2-4b22-83d2-ef5e54576651","Type":"ContainerStarted","Data":"89cdc331ebd7cedce3c8a4b3a7ab653b0d2a4a2967cde729b46f90b47ed2a971"}
Sep 09 20:33:01.035560 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:01.035433    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:33:01.035560 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:01.035498    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:33:01.037276 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:01.037251    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m86gm" event={"ID":"21bf48f4-01d2-4e62-af48-52956be58f15","Type":"ContainerStarted","Data":"a4b5386400bde46ad341e7a4758b9cc46ea67b23b9688b397a94237af66a7b12"}
Sep 09 20:33:01.051007 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:01.050985    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:33:01.051898 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:01.051883    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:33:01.065271 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:01.065235    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-25h4s" podStartSLOduration=8.717719805 podStartE2EDuration="27.06522436s" totalImagesPullingTime="18.347504555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:32:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:01.064897675 +0000 UTC m=+26.744857983" watchObservedRunningTime="2026-09-09 20:33:01.06522436 +0000 UTC m=+26.745184669"
Sep 09 20:33:01.202648 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:01.202629    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jr87x"
Sep 09 20:33:01.203208 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:01.203190    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jr87x"
Sep 09 20:33:01.885922 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:01.885884    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:33:01.886230 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:01.886002    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zpmx" podUID="14c3f4cb-c792-4945-bd65-932ed6a27aa9"
Sep 09 20:33:02.040781 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:02.040752    2556 generic.go:356] "Generic (PLEG): container finished" podID="21bf48f4-01d2-4e62-af48-52956be58f15" containerID="a4b5386400bde46ad341e7a4758b9cc46ea67b23b9688b397a94237af66a7b12" exitCode=0
Sep 09 20:33:02.041259 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:02.040823    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m86gm" event={"ID":"21bf48f4-01d2-4e62-af48-52956be58f15","Type":"ContainerDied","Data":"a4b5386400bde46ad341e7a4758b9cc46ea67b23b9688b397a94237af66a7b12"}
Sep 09 20:33:02.041259 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:02.040937    2556 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 20:33:02.488154 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:02.487977    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q2zw5"]
Sep 09 20:33:02.488279 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:02.488232    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:33:02.488356 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:02.488335    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zw5" podUID="61bb5c63-cbb7-4186-9c23-5706dba8cec2"
Sep 09 20:33:02.490738 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:02.490719    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2zpmx"]
Sep 09 20:33:02.490864 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:02.490788    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:33:02.490945 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:02.490863    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zpmx" podUID="14c3f4cb-c792-4945-bd65-932ed6a27aa9"
Sep 09 20:33:02.507351 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:02.507327    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bwnpx"]
Sep 09 20:33:02.507432 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:02.507422    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:33:02.507509 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:02.507494    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwnpx" podUID="73ad4e42-0fc1-4d62-afcf-596c76c4ea87"
Sep 09 20:33:03.047711 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:03.047680    2556 generic.go:356] "Generic (PLEG): container finished" podID="21bf48f4-01d2-4e62-af48-52956be58f15" containerID="ab1f94ba6264f37eb342aa682fc70b5a162e6c35858bddf0394b08b324f3ce63" exitCode=0
Sep 09 20:33:03.048211 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:03.047764    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m86gm" event={"ID":"21bf48f4-01d2-4e62-af48-52956be58f15","Type":"ContainerDied","Data":"ab1f94ba6264f37eb342aa682fc70b5a162e6c35858bddf0394b08b324f3ce63"}
Sep 09 20:33:03.048211 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:03.047837    2556 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 20:33:03.886218 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:03.886190    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:33:03.886345 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:03.886201    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:33:03.886345 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:03.886291    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwnpx" podUID="73ad4e42-0fc1-4d62-afcf-596c76c4ea87"
Sep 09 20:33:03.886461 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:03.886374    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zpmx" podUID="14c3f4cb-c792-4945-bd65-932ed6a27aa9"
Sep 09 20:33:03.886461 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:03.886200    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:33:03.886551 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:03.886483    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zw5" podUID="61bb5c63-cbb7-4186-9c23-5706dba8cec2"
Sep 09 20:33:04.053077 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:04.053047    2556 generic.go:356] "Generic (PLEG): container finished" podID="21bf48f4-01d2-4e62-af48-52956be58f15" containerID="ebb3498f8d4306188883a4a1eafcaf0bb656cea3768e39a354e677068f5a1583" exitCode=0
Sep 09 20:33:04.053406 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:04.053123    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m86gm" event={"ID":"21bf48f4-01d2-4e62-af48-52956be58f15","Type":"ContainerDied","Data":"ebb3498f8d4306188883a4a1eafcaf0bb656cea3768e39a354e677068f5a1583"}
Sep 09 20:33:05.255768 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:05.255735    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:33:05.256310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:05.255971    2556 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 20:33:05.266270 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:05.266232    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-25h4s" podUID="33ba58cf-b5d2-4b22-83d2-ef5e54576651" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 09 20:33:05.274988 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:05.274959    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-25h4s" podUID="33ba58cf-b5d2-4b22-83d2-ef5e54576651" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 09 20:33:05.886839 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:05.886806    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:33:05.886839 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:05.886835    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:33:05.887092 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:05.886993    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zpmx" podUID="14c3f4cb-c792-4945-bd65-932ed6a27aa9"
Sep 09 20:33:05.887092 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:05.887000    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:33:05.887092 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:05.887067    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zw5" podUID="61bb5c63-cbb7-4186-9c23-5706dba8cec2"
Sep 09 20:33:05.887240 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:05.887142    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwnpx" podUID="73ad4e42-0fc1-4d62-afcf-596c76c4ea87"
Sep 09 20:33:07.886701 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:07.886654    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:33:07.886701 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:07.886680    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:33:07.887208 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:07.886765    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zpmx" podUID="14c3f4cb-c792-4945-bd65-932ed6a27aa9"
Sep 09 20:33:07.887208 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:07.886839    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:33:07.887208 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:07.886947    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwnpx" podUID="73ad4e42-0fc1-4d62-afcf-596c76c4ea87"
Sep 09 20:33:07.887208 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:07.887030    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zw5" podUID="61bb5c63-cbb7-4186-9c23-5706dba8cec2"
Sep 09 20:33:08.358565 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:08.358529    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bmnnt_61416974-f481-4e36-ab4a-2be050c06afe/dns-node-resolver/0.log"
Sep 09 20:33:08.391901 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:08.391843    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs\") pod \"network-metrics-daemon-bwnpx\" (UID: \"73ad4e42-0fc1-4d62-afcf-596c76c4ea87\") " pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:33:08.392075 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:08.392005    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:33:08.392075 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:08.392064    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs podName:73ad4e42-0fc1-4d62-afcf-596c76c4ea87 nodeName:}" failed. No retries permitted until 2026-09-09 20:33:40.392051394 +0000 UTC m=+66.072011696 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs") pod "network-metrics-daemon-bwnpx" (UID: "73ad4e42-0fc1-4d62-afcf-596c76c4ea87") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:33:08.492833 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:08.492807    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj47h\" (UniqueName: \"kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h\") pod \"network-check-target-2zpmx\" (UID: \"14c3f4cb-c792-4945-bd65-932ed6a27aa9\") " pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:33:08.493059 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:08.492873    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret\") pod \"global-pull-secret-syncer-q2zw5\" (UID: \"61bb5c63-cbb7-4186-9c23-5706dba8cec2\") " pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:33:08.493059 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:08.492978    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:33:08.493059 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:08.492991    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:33:08.493059 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:08.493019    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:33:08.493059 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:08.493034    2556 projected.go:196] Error preparing data for projected volume kube-api-access-nj47h for pod openshift-network-diagnostics/network-check-target-2zpmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:33:08.493059 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:08.493047    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret podName:61bb5c63-cbb7-4186-9c23-5706dba8cec2 nodeName:}" failed. No retries permitted until 2026-09-09 20:33:40.493028351 +0000 UTC m=+66.172988663 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret") pod "global-pull-secret-syncer-q2zw5" (UID: "61bb5c63-cbb7-4186-9c23-5706dba8cec2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:33:08.493383 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:08.493085    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h podName:14c3f4cb-c792-4945-bd65-932ed6a27aa9 nodeName:}" failed. No retries permitted until 2026-09-09 20:33:40.493069248 +0000 UTC m=+66.173029553 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nj47h" (UniqueName: "kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h") pod "network-check-target-2zpmx" (UID: "14c3f4cb-c792-4945-bd65-932ed6a27aa9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:33:09.067301 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.067269    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeReady"
Sep 09 20:33:09.067689 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.067421    2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 20:33:09.130366 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.130339    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xxt4f"]
Sep 09 20:33:09.160819 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.160789    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nmmbf"]
Sep 09 20:33:09.160982 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.160897    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xxt4f"
Sep 09 20:33:09.163815 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.163790    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 20:33:09.163961 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.163793    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2sgfl\""
Sep 09 20:33:09.164120 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.164100    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 20:33:09.182643 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.182617    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xxt4f"]
Sep 09 20:33:09.182738 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.182658    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nmmbf"]
Sep 09 20:33:09.182738 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.182693    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vtzkg"]
Sep 09 20:33:09.182852 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.182833    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:09.185375 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.185359    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 20:33:09.185616 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.185597    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 20:33:09.185616 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.185610    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 20:33:09.185763 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.185655    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 20:33:09.185888 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.185869    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ng47s\""
Sep 09 20:33:09.200744 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.200724    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vtzkg"]
Sep 09 20:33:09.200839 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.200810    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vtzkg"
Sep 09 20:33:09.203450 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.203398    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 20:33:09.203592 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.203564    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-k7p7g\""
Sep 09 20:33:09.203687 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.203611    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 20:33:09.203687 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.203633    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 20:33:09.299584 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.299558    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/420e8660-43a1-4d78-841b-75448e999ac8-cert\") pod \"ingress-canary-vtzkg\" (UID: \"420e8660-43a1-4d78-841b-75448e999ac8\") " pod="openshift-ingress-canary/ingress-canary-vtzkg"
Sep 09 20:33:09.299696 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.299590    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-config-volume\") pod \"dns-default-xxt4f\" (UID: \"53e51d8b-4e88-46d9-9b73-d5c9b354ec0b\") " pod="openshift-dns/dns-default-xxt4f"
Sep 09 20:33:09.299696 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.299615    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53ffa66d-d8a1-445f-a1cf-14d11c651095-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:09.299696 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.299640    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-metrics-tls\") pod \"dns-default-xxt4f\" (UID: \"53e51d8b-4e88-46d9-9b73-d5c9b354ec0b\") " pod="openshift-dns/dns-default-xxt4f"
Sep 09 20:33:09.299696 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.299670    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53ffa66d-d8a1-445f-a1cf-14d11c651095-data-volume\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:09.299926 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.299712    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53ffa66d-d8a1-445f-a1cf-14d11c651095-crio-socket\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:09.299926 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.299746    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ffa66d-d8a1-445f-a1cf-14d11c651095-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:09.299926 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.299772    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrcqs\" (UniqueName: \"kubernetes.io/projected/53ffa66d-d8a1-445f-a1cf-14d11c651095-kube-api-access-lrcqs\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:09.299926 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.299825    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-tmp-dir\") pod \"dns-default-xxt4f\" (UID: \"53e51d8b-4e88-46d9-9b73-d5c9b354ec0b\") " pod="openshift-dns/dns-default-xxt4f"
Sep 09 20:33:09.299926 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.299849    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49k8w\" (UniqueName: \"kubernetes.io/projected/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-kube-api-access-49k8w\") pod \"dns-default-xxt4f\" (UID: \"53e51d8b-4e88-46d9-9b73-d5c9b354ec0b\") " pod="openshift-dns/dns-default-xxt4f"
Sep 09 20:33:09.299926 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.299898    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7lrq\" (UniqueName: \"kubernetes.io/projected/420e8660-43a1-4d78-841b-75448e999ac8-kube-api-access-g7lrq\") pod \"ingress-canary-vtzkg\" (UID: \"420e8660-43a1-4d78-841b-75448e999ac8\") " pod="openshift-ingress-canary/ingress-canary-vtzkg"
Sep 09 20:33:09.317370 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.317328    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kj4jd_c31d1698-1b9c-451e-995a-ed666693b6a2/node-ca/0.log"
Sep 09 20:33:09.400767 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.400746    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-tmp-dir\") pod \"dns-default-xxt4f\" (UID: \"53e51d8b-4e88-46d9-9b73-d5c9b354ec0b\") " pod="openshift-dns/dns-default-xxt4f"
Sep 09 20:33:09.400837 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.400771    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49k8w\" (UniqueName: \"kubernetes.io/projected/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-kube-api-access-49k8w\") pod \"dns-default-xxt4f\" (UID: \"53e51d8b-4e88-46d9-9b73-d5c9b354ec0b\") " pod="openshift-dns/dns-default-xxt4f"
Sep 09 20:33:09.400837 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.400792    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7lrq\" (UniqueName: \"kubernetes.io/projected/420e8660-43a1-4d78-841b-75448e999ac8-kube-api-access-g7lrq\") pod \"ingress-canary-vtzkg\" (UID: \"420e8660-43a1-4d78-841b-75448e999ac8\") " pod="openshift-ingress-canary/ingress-canary-vtzkg"
Sep 09 20:33:09.400837 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.400822    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/420e8660-43a1-4d78-841b-75448e999ac8-cert\") pod \"ingress-canary-vtzkg\" (UID: \"420e8660-43a1-4d78-841b-75448e999ac8\") " pod="openshift-ingress-canary/ingress-canary-vtzkg"
Sep 09 20:33:09.400986 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.400842    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-config-volume\") pod \"dns-default-xxt4f\" (UID: \"53e51d8b-4e88-46d9-9b73-d5c9b354ec0b\") " pod="openshift-dns/dns-default-xxt4f"
Sep 09 20:33:09.400986 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.400866    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53ffa66d-d8a1-445f-a1cf-14d11c651095-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:09.400986 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.400890    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-metrics-tls\") pod \"dns-default-xxt4f\" (UID: \"53e51d8b-4e88-46d9-9b73-d5c9b354ec0b\") " pod="openshift-dns/dns-default-xxt4f"
Sep 09 20:33:09.400986 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.400937    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53ffa66d-d8a1-445f-a1cf-14d11c651095-data-volume\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:09.401219 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:09.401007    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:33:09.401219 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:09.401022    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:33:09.401219 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.401019    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53ffa66d-d8a1-445f-a1cf-14d11c651095-crio-socket\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:09.401219 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:09.401073    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53ffa66d-d8a1-445f-a1cf-14d11c651095-kube-rbac-proxy-cm podName:53ffa66d-d8a1-445f-a1cf-14d11c651095 nodeName:}" failed. No retries permitted until 2026-09-09 20:33:09.901053919 +0000 UTC m=+35.581014209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53ffa66d-d8a1-445f-a1cf-14d11c651095-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nmmbf" (UID: "53ffa66d-d8a1-445f-a1cf-14d11c651095") : configmap "kube-rbac-proxy" not found
Sep 09 20:33:09.401219 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.401135    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-tmp-dir\") pod \"dns-default-xxt4f\" (UID: \"53e51d8b-4e88-46d9-9b73-d5c9b354ec0b\") " pod="openshift-dns/dns-default-xxt4f"
Sep 09 20:33:09.401219 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:09.401138    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-metrics-tls podName:53e51d8b-4e88-46d9-9b73-d5c9b354ec0b nodeName:}" failed. No retries permitted until 2026-09-09 20:33:09.901122324 +0000 UTC m=+35.581082614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-metrics-tls") pod "dns-default-xxt4f" (UID: "53e51d8b-4e88-46d9-9b73-d5c9b354ec0b") : secret "dns-default-metrics-tls" not found
Sep 09 20:33:09.401219 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.401183    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53ffa66d-d8a1-445f-a1cf-14d11c651095-crio-socket\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:09.401219 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.401205    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ffa66d-d8a1-445f-a1cf-14d11c651095-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:09.401541 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.401234    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrcqs\" (UniqueName: \"kubernetes.io/projected/53ffa66d-d8a1-445f-a1cf-14d11c651095-kube-api-access-lrcqs\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:09.401541 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:09.401338    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:33:09.401541 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.401353    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53ffa66d-d8a1-445f-a1cf-14d11c651095-data-volume\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:09.401541 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:09.401390    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ffa66d-d8a1-445f-a1cf-14d11c651095-insights-runtime-extractor-tls podName:53ffa66d-d8a1-445f-a1cf-14d11c651095 nodeName:}" failed. No retries permitted until 2026-09-09 20:33:09.901376522 +0000 UTC m=+35.581336810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53ffa66d-d8a1-445f-a1cf-14d11c651095-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nmmbf" (UID: "53ffa66d-d8a1-445f-a1cf-14d11c651095") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:33:09.401541 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.401413    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-config-volume\") pod \"dns-default-xxt4f\" (UID: \"53e51d8b-4e88-46d9-9b73-d5c9b354ec0b\") " pod="openshift-dns/dns-default-xxt4f"
Sep 09 20:33:09.404803 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.404787    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/420e8660-43a1-4d78-841b-75448e999ac8-cert\") pod \"ingress-canary-vtzkg\" (UID: \"420e8660-43a1-4d78-841b-75448e999ac8\") " pod="openshift-ingress-canary/ingress-canary-vtzkg"
Sep 09 20:33:09.412478 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.412453    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7lrq\" (UniqueName: \"kubernetes.io/projected/420e8660-43a1-4d78-841b-75448e999ac8-kube-api-access-g7lrq\") pod \"ingress-canary-vtzkg\" (UID: \"420e8660-43a1-4d78-841b-75448e999ac8\") " pod="openshift-ingress-canary/ingress-canary-vtzkg"
Sep 09 20:33:09.412693 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.412677    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49k8w\" (UniqueName: \"kubernetes.io/projected/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-kube-api-access-49k8w\") pod \"dns-default-xxt4f\" (UID: \"53e51d8b-4e88-46d9-9b73-d5c9b354ec0b\") " pod="openshift-dns/dns-default-xxt4f"
Sep 09 20:33:09.422731 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.422713    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrcqs\" (UniqueName: \"kubernetes.io/projected/53ffa66d-d8a1-445f-a1cf-14d11c651095-kube-api-access-lrcqs\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:09.511248 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.511224    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vtzkg"
Sep 09 20:33:09.807580 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.807556    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-czqvv"]
Sep 09 20:33:09.824112 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.824084    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-czqvv"
Sep 09 20:33:09.825834 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.825812    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-czqvv"]
Sep 09 20:33:09.828033 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.828013    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 09 20:33:09.828147 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.828061    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 09 20:33:09.829344 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.829320    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-lcg8g\""
Sep 09 20:33:09.829441 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.829382    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 09 20:33:09.829608 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.829590    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 09 20:33:09.886441 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.886389    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:33:09.886441 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.886412    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:33:09.886441 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.886416    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:33:09.889473 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.889458    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7r4f4\""
Sep 09 20:33:09.889699 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.889684    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:33:09.889790 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.889722    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:33:09.889790 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.889748    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9w8hr\""
Sep 09 20:33:09.889884 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.889817    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:33:09.890979 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.890962    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:33:09.905795 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.905777    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53ffa66d-d8a1-445f-a1cf-14d11c651095-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:09.905878 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.905803    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-metrics-tls\") pod \"dns-default-xxt4f\" (UID: \"53e51d8b-4e88-46d9-9b73-d5c9b354ec0b\") " pod="openshift-dns/dns-default-xxt4f"
Sep 09 20:33:09.905878 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.905821    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ffd2627f-7f96-4fc3-93c8-3f48cb930f3c-signing-key\") pod \"service-ca-75d99c484-czqvv\" (UID: \"ffd2627f-7f96-4fc3-93c8-3f48cb930f3c\") " pod="openshift-service-ca/service-ca-75d99c484-czqvv"
Sep 09 20:33:09.905972 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:09.905889    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:33:09.905972 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:09.905899    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:33:09.905972 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.905946    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ffd2627f-7f96-4fc3-93c8-3f48cb930f3c-signing-cabundle\") pod \"service-ca-75d99c484-czqvv\" (UID: \"ffd2627f-7f96-4fc3-93c8-3f48cb930f3c\") " pod="openshift-service-ca/service-ca-75d99c484-czqvv"
Sep 09 20:33:09.905972 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:09.905964    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-metrics-tls podName:53e51d8b-4e88-46d9-9b73-d5c9b354ec0b nodeName:}" failed. No retries permitted until 2026-09-09 20:33:10.905947852 +0000 UTC m=+36.585908156 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-metrics-tls") pod "dns-default-xxt4f" (UID: "53e51d8b-4e88-46d9-9b73-d5c9b354ec0b") : secret "dns-default-metrics-tls" not found
Sep 09 20:33:09.906103 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:09.906006    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53ffa66d-d8a1-445f-a1cf-14d11c651095-kube-rbac-proxy-cm podName:53ffa66d-d8a1-445f-a1cf-14d11c651095 nodeName:}" failed. No retries permitted until 2026-09-09 20:33:10.905989001 +0000 UTC m=+36.585949296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53ffa66d-d8a1-445f-a1cf-14d11c651095-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nmmbf" (UID: "53ffa66d-d8a1-445f-a1cf-14d11c651095") : configmap "kube-rbac-proxy" not found
Sep 09 20:33:09.906103 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.906019    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v55p8\" (UniqueName: \"kubernetes.io/projected/ffd2627f-7f96-4fc3-93c8-3f48cb930f3c-kube-api-access-v55p8\") pod \"service-ca-75d99c484-czqvv\" (UID: \"ffd2627f-7f96-4fc3-93c8-3f48cb930f3c\") " pod="openshift-service-ca/service-ca-75d99c484-czqvv"
Sep 09 20:33:09.906103 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.906067    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ffa66d-d8a1-445f-a1cf-14d11c651095-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:09.906196 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:09.906147    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:33:09.906196 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:09.906191    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ffa66d-d8a1-445f-a1cf-14d11c651095-insights-runtime-extractor-tls podName:53ffa66d-d8a1-445f-a1cf-14d11c651095 nodeName:}" failed. No retries permitted until 2026-09-09 20:33:10.906172306 +0000 UTC m=+36.586132614 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53ffa66d-d8a1-445f-a1cf-14d11c651095-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nmmbf" (UID: "53ffa66d-d8a1-445f-a1cf-14d11c651095") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:33:09.919567 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:09.919545    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vtzkg"]
Sep 09 20:33:09.923621 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:33:09.923592    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420e8660_43a1_4d78_841b_75448e999ac8.slice/crio-0d84abce1ed37161124b2fa9ffa427f0eb3f70baf20dd378b8875d19a60c0c5f WatchSource:0}: Error finding container 0d84abce1ed37161124b2fa9ffa427f0eb3f70baf20dd378b8875d19a60c0c5f: Status 404 returned error can't find the container with id 0d84abce1ed37161124b2fa9ffa427f0eb3f70baf20dd378b8875d19a60c0c5f
Sep 09 20:33:10.007128 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:10.007102    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ffd2627f-7f96-4fc3-93c8-3f48cb930f3c-signing-key\") pod \"service-ca-75d99c484-czqvv\" (UID: \"ffd2627f-7f96-4fc3-93c8-3f48cb930f3c\") " pod="openshift-service-ca/service-ca-75d99c484-czqvv"
Sep 09 20:33:10.007230 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:10.007132    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ffd2627f-7f96-4fc3-93c8-3f48cb930f3c-signing-cabundle\") pod \"service-ca-75d99c484-czqvv\" (UID: \"ffd2627f-7f96-4fc3-93c8-3f48cb930f3c\") " pod="openshift-service-ca/service-ca-75d99c484-czqvv"
Sep 09 20:33:10.007230 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:10.007149    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v55p8\" (UniqueName: \"kubernetes.io/projected/ffd2627f-7f96-4fc3-93c8-3f48cb930f3c-kube-api-access-v55p8\") pod \"service-ca-75d99c484-czqvv\" (UID: \"ffd2627f-7f96-4fc3-93c8-3f48cb930f3c\") " pod="openshift-service-ca/service-ca-75d99c484-czqvv"
Sep 09 20:33:10.007789 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:10.007768    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ffd2627f-7f96-4fc3-93c8-3f48cb930f3c-signing-cabundle\") pod \"service-ca-75d99c484-czqvv\" (UID: \"ffd2627f-7f96-4fc3-93c8-3f48cb930f3c\") " pod="openshift-service-ca/service-ca-75d99c484-czqvv"
Sep 09 20:33:10.009791 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:10.009772    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ffd2627f-7f96-4fc3-93c8-3f48cb930f3c-signing-key\") pod \"service-ca-75d99c484-czqvv\" (UID: \"ffd2627f-7f96-4fc3-93c8-3f48cb930f3c\") " pod="openshift-service-ca/service-ca-75d99c484-czqvv"
Sep 09 20:33:10.017009 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:10.016981    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v55p8\" (UniqueName: \"kubernetes.io/projected/ffd2627f-7f96-4fc3-93c8-3f48cb930f3c-kube-api-access-v55p8\") pod \"service-ca-75d99c484-czqvv\" (UID: \"ffd2627f-7f96-4fc3-93c8-3f48cb930f3c\") " pod="openshift-service-ca/service-ca-75d99c484-czqvv"
Sep 09 20:33:10.067411 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:10.067381    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vtzkg" event={"ID":"420e8660-43a1-4d78-841b-75448e999ac8","Type":"ContainerStarted","Data":"0d84abce1ed37161124b2fa9ffa427f0eb3f70baf20dd378b8875d19a60c0c5f"}
Sep 09 20:33:10.070239 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:10.070214    2556 generic.go:356] "Generic (PLEG): container finished" podID="21bf48f4-01d2-4e62-af48-52956be58f15" containerID="9068a0c2ffa142d898915979118aaefb683bfc0361fc2f7794d9e8d123c6cd87" exitCode=0
Sep 09 20:33:10.070312 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:10.070245    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m86gm" event={"ID":"21bf48f4-01d2-4e62-af48-52956be58f15","Type":"ContainerDied","Data":"9068a0c2ffa142d898915979118aaefb683bfc0361fc2f7794d9e8d123c6cd87"}
Sep 09 20:33:10.137953 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:10.137889    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-czqvv"
Sep 09 20:33:10.429828 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:10.429723    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-czqvv"]
Sep 09 20:33:10.432087 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:33:10.432063    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd2627f_7f96_4fc3_93c8_3f48cb930f3c.slice/crio-7d004506bcd7d2a41cb0fbed689c907f41df908ad98f1bde8c8b07f3887bfd87 WatchSource:0}: Error finding container 7d004506bcd7d2a41cb0fbed689c907f41df908ad98f1bde8c8b07f3887bfd87: Status 404 returned error can't find the container with id 7d004506bcd7d2a41cb0fbed689c907f41df908ad98f1bde8c8b07f3887bfd87
Sep 09 20:33:10.915859 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:10.915826    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ffa66d-d8a1-445f-a1cf-14d11c651095-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:10.916064 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:10.915891    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53ffa66d-d8a1-445f-a1cf-14d11c651095-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:10.916064 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:10.915939    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-metrics-tls\") pod \"dns-default-xxt4f\" (UID: \"53e51d8b-4e88-46d9-9b73-d5c9b354ec0b\") " pod="openshift-dns/dns-default-xxt4f"
Sep 09 20:33:10.916064 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:10.916003    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:33:10.916064 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:10.916051    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:33:10.916244 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:10.916071    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ffa66d-d8a1-445f-a1cf-14d11c651095-insights-runtime-extractor-tls podName:53ffa66d-d8a1-445f-a1cf-14d11c651095 nodeName:}" failed. No retries permitted until 2026-09-09 20:33:12.916049944 +0000 UTC m=+38.596010235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53ffa66d-d8a1-445f-a1cf-14d11c651095-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nmmbf" (UID: "53ffa66d-d8a1-445f-a1cf-14d11c651095") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:33:10.916244 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:10.916078    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:33:10.916244 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:10.916102    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-metrics-tls podName:53e51d8b-4e88-46d9-9b73-d5c9b354ec0b nodeName:}" failed. No retries permitted until 2026-09-09 20:33:12.916086987 +0000 UTC m=+38.596047277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-metrics-tls") pod "dns-default-xxt4f" (UID: "53e51d8b-4e88-46d9-9b73-d5c9b354ec0b") : secret "dns-default-metrics-tls" not found
Sep 09 20:33:10.916244 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:10.916130    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53ffa66d-d8a1-445f-a1cf-14d11c651095-kube-rbac-proxy-cm podName:53ffa66d-d8a1-445f-a1cf-14d11c651095 nodeName:}" failed. No retries permitted until 2026-09-09 20:33:12.916115824 +0000 UTC m=+38.596076114 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53ffa66d-d8a1-445f-a1cf-14d11c651095-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nmmbf" (UID: "53ffa66d-d8a1-445f-a1cf-14d11c651095") : configmap "kube-rbac-proxy" not found
Sep 09 20:33:11.075157 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:11.074805    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-czqvv" event={"ID":"ffd2627f-7f96-4fc3-93c8-3f48cb930f3c","Type":"ContainerStarted","Data":"7d004506bcd7d2a41cb0fbed689c907f41df908ad98f1bde8c8b07f3887bfd87"}
Sep 09 20:33:11.079334 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:11.079305    2556 generic.go:356] "Generic (PLEG): container finished" podID="21bf48f4-01d2-4e62-af48-52956be58f15" containerID="f4b7f8d880ec97463fd2c6e2123e2d95f98ae5ab68f9add76653319b63d5f8ee" exitCode=0
Sep 09 20:33:11.079446 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:11.079370    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m86gm" event={"ID":"21bf48f4-01d2-4e62-af48-52956be58f15","Type":"ContainerDied","Data":"f4b7f8d880ec97463fd2c6e2123e2d95f98ae5ab68f9add76653319b63d5f8ee"}
Sep 09 20:33:12.086404 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:12.086369    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m86gm" event={"ID":"21bf48f4-01d2-4e62-af48-52956be58f15","Type":"ContainerStarted","Data":"d0964abd37dcc54045fe22d504b53bd77877e6058ed4028d7c9fc5d624506886"}
Sep 09 20:33:12.116668 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:12.116619    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m86gm" podStartSLOduration=6.955928938 podStartE2EDuration="37.116604231s" totalImagesPullingTime="30.160675293s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:32:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:12.116116392 +0000 UTC m=+37.796076701" watchObservedRunningTime="2026-09-09 20:33:12.116604231 +0000 UTC m=+37.796564543"
Sep 09 20:33:12.933013 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:12.932977    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53ffa66d-d8a1-445f-a1cf-14d11c651095-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:12.933162 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:12.933019    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-metrics-tls\") pod \"dns-default-xxt4f\" (UID: \"53e51d8b-4e88-46d9-9b73-d5c9b354ec0b\") " pod="openshift-dns/dns-default-xxt4f"
Sep 09 20:33:12.933162 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:12.933082    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ffa66d-d8a1-445f-a1cf-14d11c651095-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:12.933162 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:12.933096    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:33:12.933162 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:12.933160    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53ffa66d-d8a1-445f-a1cf-14d11c651095-kube-rbac-proxy-cm podName:53ffa66d-d8a1-445f-a1cf-14d11c651095 nodeName:}" failed. No retries permitted until 2026-09-09 20:33:16.93314215 +0000 UTC m=+42.613102450 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53ffa66d-d8a1-445f-a1cf-14d11c651095-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nmmbf" (UID: "53ffa66d-d8a1-445f-a1cf-14d11c651095") : configmap "kube-rbac-proxy" not found
Sep 09 20:33:12.933302 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:12.933193    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:33:12.933302 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:12.933210    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:33:12.933302 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:12.933246    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-metrics-tls podName:53e51d8b-4e88-46d9-9b73-d5c9b354ec0b nodeName:}" failed. No retries permitted until 2026-09-09 20:33:16.933233175 +0000 UTC m=+42.613193465 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-metrics-tls") pod "dns-default-xxt4f" (UID: "53e51d8b-4e88-46d9-9b73-d5c9b354ec0b") : secret "dns-default-metrics-tls" not found
Sep 09 20:33:12.933302 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:12.933282    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ffa66d-d8a1-445f-a1cf-14d11c651095-insights-runtime-extractor-tls podName:53ffa66d-d8a1-445f-a1cf-14d11c651095 nodeName:}" failed. No retries permitted until 2026-09-09 20:33:16.933269505 +0000 UTC m=+42.613229795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53ffa66d-d8a1-445f-a1cf-14d11c651095-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nmmbf" (UID: "53ffa66d-d8a1-445f-a1cf-14d11c651095") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:33:14.092300 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:14.092265    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vtzkg" event={"ID":"420e8660-43a1-4d78-841b-75448e999ac8","Type":"ContainerStarted","Data":"a4fa2e180f2b34d8b3b242fc10a204131ecfe18beb34bdc9c2cd664dd265120a"}
Sep 09 20:33:14.093844 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:14.093818    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-czqvv" event={"ID":"ffd2627f-7f96-4fc3-93c8-3f48cb930f3c","Type":"ContainerStarted","Data":"22e9a5b2b1e1d222b262d76d30f6ef79b1386b0e8458b8e54fac4c071568069f"}
Sep 09 20:33:14.116421 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:14.116380    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vtzkg" podStartSLOduration=1.798738396 podStartE2EDuration="5.116367805s" totalImagesPullingTime="3.317629409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:14.116147935 +0000 UTC m=+39.796108245" watchObservedRunningTime="2026-09-09 20:33:14.116367805 +0000 UTC m=+39.796328113"
Sep 09 20:33:14.135486 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:14.135447    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-czqvv" podStartSLOduration=2.321765679 podStartE2EDuration="5.135435941s" totalImagesPullingTime="2.813670262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:14.134567183 +0000 UTC m=+39.814527492" watchObservedRunningTime="2026-09-09 20:33:14.135435941 +0000 UTC m=+39.815396250"
Sep 09 20:33:16.963736 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:16.963702    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-metrics-tls\") pod \"dns-default-xxt4f\" (UID: \"53e51d8b-4e88-46d9-9b73-d5c9b354ec0b\") " pod="openshift-dns/dns-default-xxt4f"
Sep 09 20:33:16.964178 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:16.963765    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ffa66d-d8a1-445f-a1cf-14d11c651095-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:16.964178 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:16.963840    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:33:16.964178 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:16.963846    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53ffa66d-d8a1-445f-a1cf-14d11c651095-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:16.964178 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:16.963853    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:33:16.964178 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:16.963894    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-metrics-tls podName:53e51d8b-4e88-46d9-9b73-d5c9b354ec0b nodeName:}" failed. No retries permitted until 2026-09-09 20:33:24.963879244 +0000 UTC m=+50.643839531 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-metrics-tls") pod "dns-default-xxt4f" (UID: "53e51d8b-4e88-46d9-9b73-d5c9b354ec0b") : secret "dns-default-metrics-tls" not found
Sep 09 20:33:16.964178 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:16.963924    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ffa66d-d8a1-445f-a1cf-14d11c651095-insights-runtime-extractor-tls podName:53ffa66d-d8a1-445f-a1cf-14d11c651095 nodeName:}" failed. No retries permitted until 2026-09-09 20:33:24.963901439 +0000 UTC m=+50.643861726 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53ffa66d-d8a1-445f-a1cf-14d11c651095-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nmmbf" (UID: "53ffa66d-d8a1-445f-a1cf-14d11c651095") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:33:16.964178 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:16.963941    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:33:16.964178 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:16.963991    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53ffa66d-d8a1-445f-a1cf-14d11c651095-kube-rbac-proxy-cm podName:53ffa66d-d8a1-445f-a1cf-14d11c651095 nodeName:}" failed. No retries permitted until 2026-09-09 20:33:24.963976629 +0000 UTC m=+50.643936930 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53ffa66d-d8a1-445f-a1cf-14d11c651095-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nmmbf" (UID: "53ffa66d-d8a1-445f-a1cf-14d11c651095") : configmap "kube-rbac-proxy" not found
Sep 09 20:33:25.014476 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:25.014442    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53ffa66d-d8a1-445f-a1cf-14d11c651095-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:25.014476 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:25.014476    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-metrics-tls\") pod \"dns-default-xxt4f\" (UID: \"53e51d8b-4e88-46d9-9b73-d5c9b354ec0b\") " pod="openshift-dns/dns-default-xxt4f"
Sep 09 20:33:25.015081 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:25.014512    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ffa66d-d8a1-445f-a1cf-14d11c651095-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:25.015081 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:25.014595    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:33:25.015081 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:25.014643    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ffa66d-d8a1-445f-a1cf-14d11c651095-insights-runtime-extractor-tls podName:53ffa66d-d8a1-445f-a1cf-14d11c651095 nodeName:}" failed. No retries permitted until 2026-09-09 20:33:41.014630135 +0000 UTC m=+66.694590423 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53ffa66d-d8a1-445f-a1cf-14d11c651095-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nmmbf" (UID: "53ffa66d-d8a1-445f-a1cf-14d11c651095") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:33:25.015216 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:25.015102    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53ffa66d-d8a1-445f-a1cf-14d11c651095-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:25.017648 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:25.017626    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53e51d8b-4e88-46d9-9b73-d5c9b354ec0b-metrics-tls\") pod \"dns-default-xxt4f\" (UID: \"53e51d8b-4e88-46d9-9b73-d5c9b354ec0b\") " pod="openshift-dns/dns-default-xxt4f"
Sep 09 20:33:25.071295 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:25.071271    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xxt4f"
Sep 09 20:33:25.359412 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:25.359344    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xxt4f"]
Sep 09 20:33:25.363447 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:33:25.363420    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e51d8b_4e88_46d9_9b73_d5c9b354ec0b.slice/crio-c3c9dadd4bae6ecb300a8d2bcfaad18ac78bc9f78b2b1add2e6da612db7d9722 WatchSource:0}: Error finding container c3c9dadd4bae6ecb300a8d2bcfaad18ac78bc9f78b2b1add2e6da612db7d9722: Status 404 returned error can't find the container with id c3c9dadd4bae6ecb300a8d2bcfaad18ac78bc9f78b2b1add2e6da612db7d9722
Sep 09 20:33:26.125197 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:26.125153    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xxt4f" event={"ID":"53e51d8b-4e88-46d9-9b73-d5c9b354ec0b","Type":"ContainerStarted","Data":"c3c9dadd4bae6ecb300a8d2bcfaad18ac78bc9f78b2b1add2e6da612db7d9722"}
Sep 09 20:33:27.129043 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:27.129011    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xxt4f" event={"ID":"53e51d8b-4e88-46d9-9b73-d5c9b354ec0b","Type":"ContainerStarted","Data":"138e49201240c7f17ee8556603d84464cbfab067fd0bcc566665ffbbb2717b4a"}
Sep 09 20:33:27.129364 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:27.129062    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xxt4f" event={"ID":"53e51d8b-4e88-46d9-9b73-d5c9b354ec0b","Type":"ContainerStarted","Data":"060cc33b01f01e2683c9237c274ff2fb7c535f9d235d7573ed4fd81371ed9abf"}
Sep 09 20:33:27.129364 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:27.129094    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xxt4f"
Sep 09 20:33:27.151141 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:27.151061    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xxt4f" podStartSLOduration=16.61882201 podStartE2EDuration="18.151045298s" totalImagesPullingTime="1.532223288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:27.150761885 +0000 UTC m=+52.830722196" watchObservedRunningTime="2026-09-09 20:33:27.151045298 +0000 UTC m=+52.831005608"
Sep 09 20:33:30.166462 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.166432    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-676b97c989-qctgc"]
Sep 09 20:33:30.168256 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.168241    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.171331 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.171308    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 20:33:30.171470 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.171449    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 20:33:30.171518 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.171492    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 20:33:30.172039 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.172023    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-b4pxw\""
Sep 09 20:33:30.177978 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.177960    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 20:33:30.188852 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.188833    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-676b97c989-qctgc"]
Sep 09 20:33:30.251042 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.251018    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6915fa52-02e5-4c44-894d-cb7083b1fc99-registry-tls\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.251136 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.251046    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6915fa52-02e5-4c44-894d-cb7083b1fc99-registry-certificates\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.251136 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.251065    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6915fa52-02e5-4c44-894d-cb7083b1fc99-trusted-ca\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.251136 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.251093    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6915fa52-02e5-4c44-894d-cb7083b1fc99-installation-pull-secrets\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.251260 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.251189    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jvmq\" (UniqueName: \"kubernetes.io/projected/6915fa52-02e5-4c44-894d-cb7083b1fc99-kube-api-access-4jvmq\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.251260 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.251213    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6915fa52-02e5-4c44-894d-cb7083b1fc99-ca-trust-extracted\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.251260 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.251232    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6915fa52-02e5-4c44-894d-cb7083b1fc99-image-registry-private-configuration\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.251260 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.251252    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6915fa52-02e5-4c44-894d-cb7083b1fc99-bound-sa-token\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.351844 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.351821    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jvmq\" (UniqueName: \"kubernetes.io/projected/6915fa52-02e5-4c44-894d-cb7083b1fc99-kube-api-access-4jvmq\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.351969 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.351935    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6915fa52-02e5-4c44-894d-cb7083b1fc99-ca-trust-extracted\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.351969 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.351964    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6915fa52-02e5-4c44-894d-cb7083b1fc99-image-registry-private-configuration\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.352083 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.351985    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6915fa52-02e5-4c44-894d-cb7083b1fc99-bound-sa-token\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.352083 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.352042    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6915fa52-02e5-4c44-894d-cb7083b1fc99-registry-tls\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.352083 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.352069    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6915fa52-02e5-4c44-894d-cb7083b1fc99-registry-certificates\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.352224 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.352202    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6915fa52-02e5-4c44-894d-cb7083b1fc99-trusted-ca\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.352301 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.352272    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6915fa52-02e5-4c44-894d-cb7083b1fc99-installation-pull-secrets\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.352355 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.352307    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6915fa52-02e5-4c44-894d-cb7083b1fc99-ca-trust-extracted\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.352892 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.352865    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6915fa52-02e5-4c44-894d-cb7083b1fc99-registry-certificates\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.353203 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.353180    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6915fa52-02e5-4c44-894d-cb7083b1fc99-trusted-ca\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.354447 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.354427    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6915fa52-02e5-4c44-894d-cb7083b1fc99-image-registry-private-configuration\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.354539 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.354518    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6915fa52-02e5-4c44-894d-cb7083b1fc99-registry-tls\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.354586 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.354545    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6915fa52-02e5-4c44-894d-cb7083b1fc99-installation-pull-secrets\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.366998 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.366974    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jvmq\" (UniqueName: \"kubernetes.io/projected/6915fa52-02e5-4c44-894d-cb7083b1fc99-kube-api-access-4jvmq\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.367419 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.367397    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6915fa52-02e5-4c44-894d-cb7083b1fc99-bound-sa-token\") pod \"image-registry-676b97c989-qctgc\" (UID: \"6915fa52-02e5-4c44-894d-cb7083b1fc99\") " pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.477083 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.477032    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:30.745644 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:30.745615    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-676b97c989-qctgc"]
Sep 09 20:33:30.748433 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:33:30.748406    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6915fa52_02e5_4c44_894d_cb7083b1fc99.slice/crio-787d598106e4efd8b0dc6911091662e1cf1a29b7a711d6a36e564b778b16549f WatchSource:0}: Error finding container 787d598106e4efd8b0dc6911091662e1cf1a29b7a711d6a36e564b778b16549f: Status 404 returned error can't find the container with id 787d598106e4efd8b0dc6911091662e1cf1a29b7a711d6a36e564b778b16549f
Sep 09 20:33:31.140628 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:31.140561    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-676b97c989-qctgc" event={"ID":"6915fa52-02e5-4c44-894d-cb7083b1fc99","Type":"ContainerStarted","Data":"d7b201db07841e3843653051d25fa4b69b19c405e46058041682e30162bbbc55"}
Sep 09 20:33:31.140628 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:31.140606    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-676b97c989-qctgc" event={"ID":"6915fa52-02e5-4c44-894d-cb7083b1fc99","Type":"ContainerStarted","Data":"787d598106e4efd8b0dc6911091662e1cf1a29b7a711d6a36e564b778b16549f"}
Sep 09 20:33:31.140772 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:31.140689    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:31.163300 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:31.162608    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-676b97c989-qctgc" podStartSLOduration=1.162589965 podStartE2EDuration="1.162589965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:31.159370402 +0000 UTC m=+56.839330711" watchObservedRunningTime="2026-09-09 20:33:31.162589965 +0000 UTC m=+56.842550276"
Sep 09 20:33:33.414988 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:33.414957    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f75md"]
Sep 09 20:33:33.418407 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:33.418389    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f75md"
Sep 09 20:33:33.421192 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:33.421172    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 09 20:33:33.421300 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:33.421281    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-xlmkv\""
Sep 09 20:33:33.427396 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:33.427376    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f75md"]
Sep 09 20:33:33.475122 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:33.475096    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c687fc88-83ce-40e9-92bf-4a9802d1e021-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-f75md\" (UID: \"c687fc88-83ce-40e9-92bf-4a9802d1e021\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f75md"
Sep 09 20:33:33.576259 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:33.576233    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c687fc88-83ce-40e9-92bf-4a9802d1e021-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-f75md\" (UID: \"c687fc88-83ce-40e9-92bf-4a9802d1e021\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f75md"
Sep 09 20:33:33.576401 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:33.576385    2556 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 09 20:33:33.576480 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:33.576467    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c687fc88-83ce-40e9-92bf-4a9802d1e021-tls-certificates podName:c687fc88-83ce-40e9-92bf-4a9802d1e021 nodeName:}" failed. No retries permitted until 2026-09-09 20:33:34.076447714 +0000 UTC m=+59.756408009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/c687fc88-83ce-40e9-92bf-4a9802d1e021-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-f75md" (UID: "c687fc88-83ce-40e9-92bf-4a9802d1e021") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 09 20:33:34.080478 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:34.080445    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c687fc88-83ce-40e9-92bf-4a9802d1e021-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-f75md\" (UID: \"c687fc88-83ce-40e9-92bf-4a9802d1e021\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f75md"
Sep 09 20:33:34.082784 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:34.082753    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c687fc88-83ce-40e9-92bf-4a9802d1e021-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-f75md\" (UID: \"c687fc88-83ce-40e9-92bf-4a9802d1e021\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f75md"
Sep 09 20:33:34.327007 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:34.326982    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f75md"
Sep 09 20:33:34.590393 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:34.590324    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f75md"]
Sep 09 20:33:34.593307 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:33:34.593283    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc687fc88_83ce_40e9_92bf_4a9802d1e021.slice/crio-875a684a474242903b2524755450ab9c55299cf062433bfec2febcd36a99e508 WatchSource:0}: Error finding container 875a684a474242903b2524755450ab9c55299cf062433bfec2febcd36a99e508: Status 404 returned error can't find the container with id 875a684a474242903b2524755450ab9c55299cf062433bfec2febcd36a99e508
Sep 09 20:33:35.151531 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:35.151490    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f75md" event={"ID":"c687fc88-83ce-40e9-92bf-4a9802d1e021","Type":"ContainerStarted","Data":"875a684a474242903b2524755450ab9c55299cf062433bfec2febcd36a99e508"}
Sep 09 20:33:35.277407 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:35.277368    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-25h4s"
Sep 09 20:33:36.155253 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.155184    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f75md" event={"ID":"c687fc88-83ce-40e9-92bf-4a9802d1e021","Type":"ContainerStarted","Data":"7c176bb8c057dfeffd1950ca98b92687196e994d3beff96ea862e75ebad164e8"}
Sep 09 20:33:36.155601 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.155311    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f75md"
Sep 09 20:33:36.159986 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.159966    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f75md"
Sep 09 20:33:36.172516 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.172473    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f75md" podStartSLOduration=2.02427629 podStartE2EDuration="3.172460544s" totalImagesPullingTime="1.148184254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:36.171661205 +0000 UTC m=+61.851621526" watchObservedRunningTime="2026-09-09 20:33:36.172460544 +0000 UTC m=+61.852420853"
Sep 09 20:33:36.472600 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.472522    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-jlrwd"]
Sep 09 20:33:36.475225 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.475204    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-jlrwd"
Sep 09 20:33:36.478246 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.478221    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 20:33:36.478346 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.478280    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 09 20:33:36.478346 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.478338    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 09 20:33:36.478463 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.478356    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 20:33:36.478463 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.478280    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 20:33:36.478463 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.478397    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gkljx\""
Sep 09 20:33:36.490488 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.488307    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-jlrwd"]
Sep 09 20:33:36.501725 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.501703    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4201a6a-7846-48fb-801f-4775039db982-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-jlrwd\" (UID: \"a4201a6a-7846-48fb-801f-4775039db982\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jlrwd"
Sep 09 20:33:36.501823 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.501744    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4201a6a-7846-48fb-801f-4775039db982-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-jlrwd\" (UID: \"a4201a6a-7846-48fb-801f-4775039db982\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jlrwd"
Sep 09 20:33:36.501823 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.501806    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr27q\" (UniqueName: \"kubernetes.io/projected/a4201a6a-7846-48fb-801f-4775039db982-kube-api-access-sr27q\") pod \"prometheus-operator-6f4946db56-jlrwd\" (UID: \"a4201a6a-7846-48fb-801f-4775039db982\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jlrwd"
Sep 09 20:33:36.501959 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.501862    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a4201a6a-7846-48fb-801f-4775039db982-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-jlrwd\" (UID: \"a4201a6a-7846-48fb-801f-4775039db982\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jlrwd"
Sep 09 20:33:36.602309 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.602281    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4201a6a-7846-48fb-801f-4775039db982-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-jlrwd\" (UID: \"a4201a6a-7846-48fb-801f-4775039db982\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jlrwd"
Sep 09 20:33:36.602391 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.602323    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4201a6a-7846-48fb-801f-4775039db982-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-jlrwd\" (UID: \"a4201a6a-7846-48fb-801f-4775039db982\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jlrwd"
Sep 09 20:33:36.602391 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.602380    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr27q\" (UniqueName: \"kubernetes.io/projected/a4201a6a-7846-48fb-801f-4775039db982-kube-api-access-sr27q\") pod \"prometheus-operator-6f4946db56-jlrwd\" (UID: \"a4201a6a-7846-48fb-801f-4775039db982\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jlrwd"
Sep 09 20:33:36.602486 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:36.602430    2556 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 09 20:33:36.602537 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:36.602497    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4201a6a-7846-48fb-801f-4775039db982-prometheus-operator-tls podName:a4201a6a-7846-48fb-801f-4775039db982 nodeName:}" failed. No retries permitted until 2026-09-09 20:33:37.102477631 +0000 UTC m=+62.782437938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/a4201a6a-7846-48fb-801f-4775039db982-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-jlrwd" (UID: "a4201a6a-7846-48fb-801f-4775039db982") : secret "prometheus-operator-tls" not found
Sep 09 20:33:36.602537 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.602429    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a4201a6a-7846-48fb-801f-4775039db982-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-jlrwd\" (UID: \"a4201a6a-7846-48fb-801f-4775039db982\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jlrwd"
Sep 09 20:33:36.603132 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.603112    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4201a6a-7846-48fb-801f-4775039db982-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-jlrwd\" (UID: \"a4201a6a-7846-48fb-801f-4775039db982\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jlrwd"
Sep 09 20:33:36.604747 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.604719    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a4201a6a-7846-48fb-801f-4775039db982-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-jlrwd\" (UID: \"a4201a6a-7846-48fb-801f-4775039db982\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jlrwd"
Sep 09 20:33:36.612994 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:36.612971    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr27q\" (UniqueName: \"kubernetes.io/projected/a4201a6a-7846-48fb-801f-4775039db982-kube-api-access-sr27q\") pod \"prometheus-operator-6f4946db56-jlrwd\" (UID: \"a4201a6a-7846-48fb-801f-4775039db982\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jlrwd"
Sep 09 20:33:37.106778 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:37.106748    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4201a6a-7846-48fb-801f-4775039db982-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-jlrwd\" (UID: \"a4201a6a-7846-48fb-801f-4775039db982\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jlrwd"
Sep 09 20:33:37.109117 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:37.109085    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4201a6a-7846-48fb-801f-4775039db982-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-jlrwd\" (UID: \"a4201a6a-7846-48fb-801f-4775039db982\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-jlrwd"
Sep 09 20:33:37.133611 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:37.133586    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xxt4f"
Sep 09 20:33:37.383725 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:37.383643    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-jlrwd"
Sep 09 20:33:37.650887 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:37.650821    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-jlrwd"]
Sep 09 20:33:37.654157 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:33:37.654130    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4201a6a_7846_48fb_801f_4775039db982.slice/crio-7f774386efbd14c08a2da132688e5f7fb9c02323dd73bb998e9e3c3ae15e1bba WatchSource:0}: Error finding container 7f774386efbd14c08a2da132688e5f7fb9c02323dd73bb998e9e3c3ae15e1bba: Status 404 returned error can't find the container with id 7f774386efbd14c08a2da132688e5f7fb9c02323dd73bb998e9e3c3ae15e1bba
Sep 09 20:33:38.163007 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:38.162978    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-jlrwd" event={"ID":"a4201a6a-7846-48fb-801f-4775039db982","Type":"ContainerStarted","Data":"7f774386efbd14c08a2da132688e5f7fb9c02323dd73bb998e9e3c3ae15e1bba"}
Sep 09 20:33:40.170707 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:40.170677    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-jlrwd" event={"ID":"a4201a6a-7846-48fb-801f-4775039db982","Type":"ContainerStarted","Data":"51eb99b1ade04f546b2fefe411df31b93e50a187cdf686cdcbfda579147e6ed4"}
Sep 09 20:33:40.171109 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:40.170725    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-jlrwd" event={"ID":"a4201a6a-7846-48fb-801f-4775039db982","Type":"ContainerStarted","Data":"d295254e704d70e7ee5967d1ec4fd24b0aa32cef2c90f829718e27921bb8d160"}
Sep 09 20:33:40.191004 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:40.190950    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-jlrwd" podStartSLOduration=2.376764331 podStartE2EDuration="4.190686337s" totalImagesPullingTime="1.813922006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:40.186563421 +0000 UTC m=+65.866523731" watchObservedRunningTime="2026-09-09 20:33:40.190686337 +0000 UTC m=+65.870646629"
Sep 09 20:33:40.429706 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:40.429625    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs\") pod \"network-metrics-daemon-bwnpx\" (UID: \"73ad4e42-0fc1-4d62-afcf-596c76c4ea87\") " pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:33:40.432315 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:40.432295    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:33:40.442620 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:40.442600    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ad4e42-0fc1-4d62-afcf-596c76c4ea87-metrics-certs\") pod \"network-metrics-daemon-bwnpx\" (UID: \"73ad4e42-0fc1-4d62-afcf-596c76c4ea87\") " pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:33:40.530792 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:40.530765    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret\") pod \"global-pull-secret-syncer-q2zw5\" (UID: \"61bb5c63-cbb7-4186-9c23-5706dba8cec2\") " pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:33:40.530899 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:40.530814    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj47h\" (UniqueName: \"kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h\") pod \"network-check-target-2zpmx\" (UID: \"14c3f4cb-c792-4945-bd65-932ed6a27aa9\") " pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:33:40.533287 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:40.533267    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:33:40.533376 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:40.533323    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:33:40.540393 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:40.540377    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9w8hr\""
Sep 09 20:33:40.543406 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:40.543388    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:33:40.543482 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:40.543394    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61bb5c63-cbb7-4186-9c23-5706dba8cec2-original-pull-secret\") pod \"global-pull-secret-syncer-q2zw5\" (UID: \"61bb5c63-cbb7-4186-9c23-5706dba8cec2\") " pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:33:40.548792 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:40.548776    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwnpx"
Sep 09 20:33:40.554001 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:40.553981    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj47h\" (UniqueName: \"kubernetes.io/projected/14c3f4cb-c792-4945-bd65-932ed6a27aa9-kube-api-access-nj47h\") pod \"network-check-target-2zpmx\" (UID: \"14c3f4cb-c792-4945-bd65-932ed6a27aa9\") " pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:33:40.829162 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:40.829127    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7r4f4\""
Sep 09 20:33:40.832140 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:40.832124    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zw5"
Sep 09 20:33:40.835856 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:40.835796    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bwnpx"]
Sep 09 20:33:40.837196 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:40.837176    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:33:40.840389 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:33:40.840291    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ad4e42_0fc1_4d62_afcf_596c76c4ea87.slice/crio-6098281cb463afd53501c6f6f0f7fdccc4d36d78c3474f57e4385f9e1ecdaba6 WatchSource:0}: Error finding container 6098281cb463afd53501c6f6f0f7fdccc4d36d78c3474f57e4385f9e1ecdaba6: Status 404 returned error can't find the container with id 6098281cb463afd53501c6f6f0f7fdccc4d36d78c3474f57e4385f9e1ecdaba6
Sep 09 20:33:41.034743 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.034711    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ffa66d-d8a1-445f-a1cf-14d11c651095-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:41.037037 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.037008    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ffa66d-d8a1-445f-a1cf-14d11c651095-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nmmbf\" (UID: \"53ffa66d-d8a1-445f-a1cf-14d11c651095\") " pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:41.106741 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.106681    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q2zw5"]
Sep 09 20:33:41.109797 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:33:41.109772    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61bb5c63_cbb7_4186_9c23_5706dba8cec2.slice/crio-e460debff4b8cc1dea33b966873246ac7501b24e8707173423670555039cc73c WatchSource:0}: Error finding container e460debff4b8cc1dea33b966873246ac7501b24e8707173423670555039cc73c: Status 404 returned error can't find the container with id e460debff4b8cc1dea33b966873246ac7501b24e8707173423670555039cc73c
Sep 09 20:33:41.110865 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.110846    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2zpmx"]
Sep 09 20:33:41.113302 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:33:41.113278    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c3f4cb_c792_4945_bd65_932ed6a27aa9.slice/crio-3b73bc4785229ba15e41a11dc467a01d2c28e4b84bc7cb9c04d244b157e8622d WatchSource:0}: Error finding container 3b73bc4785229ba15e41a11dc467a01d2c28e4b84bc7cb9c04d244b157e8622d: Status 404 returned error can't find the container with id 3b73bc4785229ba15e41a11dc467a01d2c28e4b84bc7cb9c04d244b157e8622d
Sep 09 20:33:41.176791 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.176765    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q2zw5" event={"ID":"61bb5c63-cbb7-4186-9c23-5706dba8cec2","Type":"ContainerStarted","Data":"e460debff4b8cc1dea33b966873246ac7501b24e8707173423670555039cc73c"}
Sep 09 20:33:41.177983 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.177956    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bwnpx" event={"ID":"73ad4e42-0fc1-4d62-afcf-596c76c4ea87","Type":"ContainerStarted","Data":"6098281cb463afd53501c6f6f0f7fdccc4d36d78c3474f57e4385f9e1ecdaba6"}
Sep 09 20:33:41.179110 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.179091    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2zpmx" event={"ID":"14c3f4cb-c792-4945-bd65-932ed6a27aa9","Type":"ContainerStarted","Data":"3b73bc4785229ba15e41a11dc467a01d2c28e4b84bc7cb9c04d244b157e8622d"}
Sep 09 20:33:41.295059 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.295037    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ng47s\""
Sep 09 20:33:41.302580 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.302554    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nmmbf"
Sep 09 20:33:41.586659 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.586630    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nmmbf"]
Sep 09 20:33:41.828922 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.828573    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-mv87t"]
Sep 09 20:33:41.832096 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.831232    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mv87t"
Sep 09 20:33:41.835229 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.834501    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 09 20:33:41.835229 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.834766    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 09 20:33:41.835229 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.835046    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-2qjkm\""
Sep 09 20:33:41.848453 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.848414    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-mv87t"]
Sep 09 20:33:41.854315 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.854289    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cxbm6"]
Sep 09 20:33:41.860055 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.860034    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"]
Sep 09 20:33:41.860349 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.860333    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:41.864077 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.864061    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"
Sep 09 20:33:41.866015 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.864753    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 20:33:41.866015 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.864962    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 20:33:41.866015 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.865192    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-97pg4\""
Sep 09 20:33:41.866015 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.865400    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 20:33:41.873540 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.873324    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"]
Sep 09 20:33:41.875170 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.874057    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-h97lr\""
Sep 09 20:33:41.875170 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.874072    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 09 20:33:41.875170 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.874233    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 09 20:33:41.875170 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.874304    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 09 20:33:41.944423 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.943971    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/21002114-9907-455c-9324-a5ccb05b9516-node-exporter-wtmp\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:41.944537 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.944475    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c5ebe346-482a-410a-80b8-7b0a2981a38c-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-4fwvm\" (UID: \"c5ebe346-482a-410a-80b8-7b0a2981a38c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"
Sep 09 20:33:41.944537 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.944510    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21002114-9907-455c-9324-a5ccb05b9516-metrics-client-ca\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:41.944651 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.944541    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5ebe346-482a-410a-80b8-7b0a2981a38c-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-4fwvm\" (UID: \"c5ebe346-482a-410a-80b8-7b0a2981a38c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"
Sep 09 20:33:41.944651 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.944582    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5ebe346-482a-410a-80b8-7b0a2981a38c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-4fwvm\" (UID: \"c5ebe346-482a-410a-80b8-7b0a2981a38c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"
Sep 09 20:33:41.944651 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.944610    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2a295c8-1860-4315-ab35-764265d474e9-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-mv87t\" (UID: \"f2a295c8-1860-4315-ab35-764265d474e9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mv87t"
Sep 09 20:33:41.944802 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.944651    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzzgb\" (UniqueName: \"kubernetes.io/projected/f2a295c8-1860-4315-ab35-764265d474e9-kube-api-access-kzzgb\") pod \"openshift-state-metrics-657fd6c54-mv87t\" (UID: \"f2a295c8-1860-4315-ab35-764265d474e9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mv87t"
Sep 09 20:33:41.944802 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.944681    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2a295c8-1860-4315-ab35-764265d474e9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-mv87t\" (UID: \"f2a295c8-1860-4315-ab35-764265d474e9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mv87t"
Sep 09 20:33:41.944802 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.944715    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c5ebe346-482a-410a-80b8-7b0a2981a38c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-4fwvm\" (UID: \"c5ebe346-482a-410a-80b8-7b0a2981a38c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"
Sep 09 20:33:41.944802 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.944748    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21002114-9907-455c-9324-a5ccb05b9516-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:41.944802 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.944781    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21002114-9907-455c-9324-a5ccb05b9516-sys\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:41.945073 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.944804    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/21002114-9907-455c-9324-a5ccb05b9516-node-exporter-textfile\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:41.945073 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.944832    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/21002114-9907-455c-9324-a5ccb05b9516-root\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:41.945073 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.944861    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2a295c8-1860-4315-ab35-764265d474e9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-mv87t\" (UID: \"f2a295c8-1860-4315-ab35-764265d474e9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mv87t"
Sep 09 20:33:41.945271 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.945110    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z4jf\" (UniqueName: \"kubernetes.io/projected/21002114-9907-455c-9324-a5ccb05b9516-kube-api-access-8z4jf\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:41.945271 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.945157    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx8qn\" (UniqueName: \"kubernetes.io/projected/c5ebe346-482a-410a-80b8-7b0a2981a38c-kube-api-access-jx8qn\") pod \"kube-state-metrics-74cb57df7-4fwvm\" (UID: \"c5ebe346-482a-410a-80b8-7b0a2981a38c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"
Sep 09 20:33:41.945271 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.945186    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/21002114-9907-455c-9324-a5ccb05b9516-node-exporter-tls\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:41.945271 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.945212    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/21002114-9907-455c-9324-a5ccb05b9516-node-exporter-accelerators-collector-config\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:41.945271 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:41.945236    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5ebe346-482a-410a-80b8-7b0a2981a38c-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-4fwvm\" (UID: \"c5ebe346-482a-410a-80b8-7b0a2981a38c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"
Sep 09 20:33:42.047673 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.046471    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2a295c8-1860-4315-ab35-764265d474e9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-mv87t\" (UID: \"f2a295c8-1860-4315-ab35-764265d474e9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mv87t"
Sep 09 20:33:42.047673 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.046512    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c5ebe346-482a-410a-80b8-7b0a2981a38c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-4fwvm\" (UID: \"c5ebe346-482a-410a-80b8-7b0a2981a38c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"
Sep 09 20:33:42.047673 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.046547    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21002114-9907-455c-9324-a5ccb05b9516-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:42.047673 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.046573    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21002114-9907-455c-9324-a5ccb05b9516-sys\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:42.047673 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.046598    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/21002114-9907-455c-9324-a5ccb05b9516-node-exporter-textfile\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:42.047673 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.046623    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/21002114-9907-455c-9324-a5ccb05b9516-root\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:42.047673 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.046652    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2a295c8-1860-4315-ab35-764265d474e9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-mv87t\" (UID: \"f2a295c8-1860-4315-ab35-764265d474e9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mv87t"
Sep 09 20:33:42.047673 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.046700    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z4jf\" (UniqueName: \"kubernetes.io/projected/21002114-9907-455c-9324-a5ccb05b9516-kube-api-access-8z4jf\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:42.047673 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.046723    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx8qn\" (UniqueName: \"kubernetes.io/projected/c5ebe346-482a-410a-80b8-7b0a2981a38c-kube-api-access-jx8qn\") pod \"kube-state-metrics-74cb57df7-4fwvm\" (UID: \"c5ebe346-482a-410a-80b8-7b0a2981a38c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"
Sep 09 20:33:42.047673 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.046751    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/21002114-9907-455c-9324-a5ccb05b9516-node-exporter-tls\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:42.047673 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.046778    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/21002114-9907-455c-9324-a5ccb05b9516-node-exporter-accelerators-collector-config\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:42.047673 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.046801    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5ebe346-482a-410a-80b8-7b0a2981a38c-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-4fwvm\" (UID: \"c5ebe346-482a-410a-80b8-7b0a2981a38c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"
Sep 09 20:33:42.047673 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.046838    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/21002114-9907-455c-9324-a5ccb05b9516-node-exporter-wtmp\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:42.047673 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.046867    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c5ebe346-482a-410a-80b8-7b0a2981a38c-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-4fwvm\" (UID: \"c5ebe346-482a-410a-80b8-7b0a2981a38c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"
Sep 09 20:33:42.047673 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.046893    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21002114-9907-455c-9324-a5ccb05b9516-metrics-client-ca\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:42.047673 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.046932    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5ebe346-482a-410a-80b8-7b0a2981a38c-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-4fwvm\" (UID: \"c5ebe346-482a-410a-80b8-7b0a2981a38c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"
Sep 09 20:33:42.048747 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.046970    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5ebe346-482a-410a-80b8-7b0a2981a38c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-4fwvm\" (UID: \"c5ebe346-482a-410a-80b8-7b0a2981a38c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"
Sep 09 20:33:42.048747 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.046997    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2a295c8-1860-4315-ab35-764265d474e9-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-mv87t\" (UID: \"f2a295c8-1860-4315-ab35-764265d474e9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mv87t"
Sep 09 20:33:42.048747 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.047032    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzzgb\" (UniqueName: \"kubernetes.io/projected/f2a295c8-1860-4315-ab35-764265d474e9-kube-api-access-kzzgb\") pod \"openshift-state-metrics-657fd6c54-mv87t\" (UID: \"f2a295c8-1860-4315-ab35-764265d474e9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mv87t"
Sep 09 20:33:42.048747 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.047969    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/21002114-9907-455c-9324-a5ccb05b9516-node-exporter-textfile\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:42.048747 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.048245    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/21002114-9907-455c-9324-a5ccb05b9516-node-exporter-accelerators-collector-config\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:42.048747 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.048329    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c5ebe346-482a-410a-80b8-7b0a2981a38c-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-4fwvm\" (UID: \"c5ebe346-482a-410a-80b8-7b0a2981a38c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"
Sep 09 20:33:42.048747 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:42.048355    2556 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 09 20:33:42.048747 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:42.048404    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5ebe346-482a-410a-80b8-7b0a2981a38c-kube-state-metrics-tls podName:c5ebe346-482a-410a-80b8-7b0a2981a38c nodeName:}" failed. No retries permitted until 2026-09-09 20:33:42.548386829 +0000 UTC m=+68.228347134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c5ebe346-482a-410a-80b8-7b0a2981a38c-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-4fwvm" (UID: "c5ebe346-482a-410a-80b8-7b0a2981a38c") : secret "kube-state-metrics-tls" not found
Sep 09 20:33:42.048747 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.048694    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/21002114-9907-455c-9324-a5ccb05b9516-node-exporter-wtmp\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:42.049206 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:42.048797    2556 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 09 20:33:42.049206 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:42.048842    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2a295c8-1860-4315-ab35-764265d474e9-openshift-state-metrics-tls podName:f2a295c8-1860-4315-ab35-764265d474e9 nodeName:}" failed. No retries permitted until 2026-09-09 20:33:42.548826466 +0000 UTC m=+68.228786757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f2a295c8-1860-4315-ab35-764265d474e9-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-mv87t" (UID: "f2a295c8-1860-4315-ab35-764265d474e9") : secret "openshift-state-metrics-tls" not found
Sep 09 20:33:42.049206 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.048885    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/21002114-9907-455c-9324-a5ccb05b9516-root\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:42.049206 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.048994    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c5ebe346-482a-410a-80b8-7b0a2981a38c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-4fwvm\" (UID: \"c5ebe346-482a-410a-80b8-7b0a2981a38c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"
Sep 09 20:33:42.049206 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.049063    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21002114-9907-455c-9324-a5ccb05b9516-sys\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:42.049861 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.049837    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21002114-9907-455c-9324-a5ccb05b9516-metrics-client-ca\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:42.050357 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.050331    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5ebe346-482a-410a-80b8-7b0a2981a38c-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-4fwvm\" (UID: \"c5ebe346-482a-410a-80b8-7b0a2981a38c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"
Sep 09 20:33:42.050553 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.050526    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2a295c8-1860-4315-ab35-764265d474e9-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-mv87t\" (UID: \"f2a295c8-1860-4315-ab35-764265d474e9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mv87t"
Sep 09 20:33:42.053385 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.053360    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5ebe346-482a-410a-80b8-7b0a2981a38c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-4fwvm\" (UID: \"c5ebe346-482a-410a-80b8-7b0a2981a38c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"
Sep 09 20:33:42.054605 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.054557    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2a295c8-1860-4315-ab35-764265d474e9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-mv87t\" (UID: \"f2a295c8-1860-4315-ab35-764265d474e9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mv87t"
Sep 09 20:33:42.055081 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.055030    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/21002114-9907-455c-9324-a5ccb05b9516-node-exporter-tls\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:42.057062 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.057024    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21002114-9907-455c-9324-a5ccb05b9516-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:42.064994 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.064953    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzzgb\" (UniqueName: \"kubernetes.io/projected/f2a295c8-1860-4315-ab35-764265d474e9-kube-api-access-kzzgb\") pod \"openshift-state-metrics-657fd6c54-mv87t\" (UID: \"f2a295c8-1860-4315-ab35-764265d474e9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mv87t"
Sep 09 20:33:42.070551 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.070511    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx8qn\" (UniqueName: \"kubernetes.io/projected/c5ebe346-482a-410a-80b8-7b0a2981a38c-kube-api-access-jx8qn\") pod \"kube-state-metrics-74cb57df7-4fwvm\" (UID: \"c5ebe346-482a-410a-80b8-7b0a2981a38c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"
Sep 09 20:33:42.071691 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.071646    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z4jf\" (UniqueName: \"kubernetes.io/projected/21002114-9907-455c-9324-a5ccb05b9516-kube-api-access-8z4jf\") pod \"node-exporter-cxbm6\" (UID: \"21002114-9907-455c-9324-a5ccb05b9516\") " pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:42.182489 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.182456    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cxbm6"
Sep 09 20:33:42.192347 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.191603    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bwnpx" event={"ID":"73ad4e42-0fc1-4d62-afcf-596c76c4ea87","Type":"ContainerStarted","Data":"4887dda1842e13cfa566d102d7c7190a16c804841529e45f4a7d11305c268a3f"}
Sep 09 20:33:42.192347 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.191694    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bwnpx" event={"ID":"73ad4e42-0fc1-4d62-afcf-596c76c4ea87","Type":"ContainerStarted","Data":"03a12a17d61ff561f3c294d8bde69f1995e06fb5c4b646ee3937ccb199d7dc17"}
Sep 09 20:33:42.195772 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.194869    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nmmbf" event={"ID":"53ffa66d-d8a1-445f-a1cf-14d11c651095","Type":"ContainerStarted","Data":"5a6dacff3f7e8ed8f5dfb8502f9434e9bdf2e0e17bf5fdff7fee480503d2f12b"}
Sep 09 20:33:42.195772 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.194965    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nmmbf" event={"ID":"53ffa66d-d8a1-445f-a1cf-14d11c651095","Type":"ContainerStarted","Data":"b45780bed682922a10fd7c8681a02d737911e623008c2004d964656e4e619be1"}
Sep 09 20:33:42.208568 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.208515    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bwnpx" podStartSLOduration=67.150690045 podStartE2EDuration="1m8.208499103s" totalImagesPullingTime="1.057809058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:32:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:42.208453375 +0000 UTC m=+67.888413686" watchObservedRunningTime="2026-09-09 20:33:42.208499103 +0000 UTC m=+67.888459412"
Sep 09 20:33:42.552010 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.551979    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2a295c8-1860-4315-ab35-764265d474e9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-mv87t\" (UID: \"f2a295c8-1860-4315-ab35-764265d474e9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mv87t"
Sep 09 20:33:42.552142 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.552049    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5ebe346-482a-410a-80b8-7b0a2981a38c-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-4fwvm\" (UID: \"c5ebe346-482a-410a-80b8-7b0a2981a38c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"
Sep 09 20:33:42.556812 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.556784    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5ebe346-482a-410a-80b8-7b0a2981a38c-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-4fwvm\" (UID: \"c5ebe346-482a-410a-80b8-7b0a2981a38c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"
Sep 09 20:33:42.557388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.557366    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2a295c8-1860-4315-ab35-764265d474e9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-mv87t\" (UID: \"f2a295c8-1860-4315-ab35-764265d474e9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mv87t"
Sep 09 20:33:42.750478 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.750405    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mv87t"
Sep 09 20:33:42.790780 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.790710    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"
Sep 09 20:33:42.937955 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.937922    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:33:42.942478 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.942454    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:42.945514 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.945288    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 20:33:42.945514 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.945424    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8j7w6\""
Sep 09 20:33:42.945514 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.945502    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 20:33:42.946114 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.945959    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 20:33:42.946114 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.946015    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 20:33:42.946278 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.946173    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 20:33:42.946278 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.946209    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 20:33:42.946278 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.946251    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 20:33:42.946528 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.946512    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 20:33:42.946757 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.946740    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 20:33:42.958962 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:42.958940    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:33:43.043958 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.042149    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-mv87t"]
Sep 09 20:33:43.056207 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.056183    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e972a506-9e4a-44dd-8924-1e56ad707ac6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.056310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.056233    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.056310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.056265    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e972a506-9e4a-44dd-8924-1e56ad707ac6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.056310 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.056295    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.056464 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.056343    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e972a506-9e4a-44dd-8924-1e56ad707ac6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.056464 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.056371    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.056464 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.056399    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zdk9\" (UniqueName: \"kubernetes.io/projected/e972a506-9e4a-44dd-8924-1e56ad707ac6-kube-api-access-5zdk9\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.056464 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.056431    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e972a506-9e4a-44dd-8924-1e56ad707ac6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.056464 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.056462    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-web-config\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.056716 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.056492    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-config-volume\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.056716 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.056513    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.056716 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.056552    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e972a506-9e4a-44dd-8924-1e56ad707ac6-config-out\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.056716 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.056583    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.091088 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.091038    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm"]
Sep 09 20:33:43.097685 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:33:43.097653    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ebe346_482a_410a_80b8_7b0a2981a38c.slice/crio-7316351cfacc5dfd9938a32072b74be1bc433202c135d2465736d4976317e69f WatchSource:0}: Error finding container 7316351cfacc5dfd9938a32072b74be1bc433202c135d2465736d4976317e69f: Status 404 returned error can't find the container with id 7316351cfacc5dfd9938a32072b74be1bc433202c135d2465736d4976317e69f
Sep 09 20:33:43.158098 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.157067    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e972a506-9e4a-44dd-8924-1e56ad707ac6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.158098 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.157115    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.158098 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.157146    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e972a506-9e4a-44dd-8924-1e56ad707ac6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.158098 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.157174    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.158098 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.157221    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e972a506-9e4a-44dd-8924-1e56ad707ac6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.158098 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.157252    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.158098 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.157280    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zdk9\" (UniqueName: \"kubernetes.io/projected/e972a506-9e4a-44dd-8924-1e56ad707ac6-kube-api-access-5zdk9\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.158098 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.157307    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e972a506-9e4a-44dd-8924-1e56ad707ac6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.158098 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.157335    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-web-config\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.158098 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.157360    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-config-volume\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.158098 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.157385    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.158098 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.157415    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e972a506-9e4a-44dd-8924-1e56ad707ac6-config-out\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.158098 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.157439    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.161200 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:33:43.159245    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e972a506-9e4a-44dd-8924-1e56ad707ac6-alertmanager-trusted-ca-bundle podName:e972a506-9e4a-44dd-8924-1e56ad707ac6 nodeName:}" failed. No retries permitted until 2026-09-09 20:33:43.65921984 +0000 UTC m=+69.339180142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/e972a506-9e4a-44dd-8924-1e56ad707ac6-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "e972a506-9e4a-44dd-8924-1e56ad707ac6") : configmap references non-existent config key: ca-bundle.crt
Sep 09 20:33:43.164145 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.163698    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e972a506-9e4a-44dd-8924-1e56ad707ac6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.164734 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.164611    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-config-volume\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.165468 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.165401    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e972a506-9e4a-44dd-8924-1e56ad707ac6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.167319 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.165726    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.167319 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.166201    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.167319 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.166543    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e972a506-9e4a-44dd-8924-1e56ad707ac6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.167319 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.166671    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.167319 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.166932    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-web-config\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.167319 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.167287    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.168344 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.168116    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.168344 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.168289    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e972a506-9e4a-44dd-8924-1e56ad707ac6-config-out\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.179027 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.178968    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zdk9\" (UniqueName: \"kubernetes.io/projected/e972a506-9e4a-44dd-8924-1e56ad707ac6-kube-api-access-5zdk9\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.204093 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.203509    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mv87t" event={"ID":"f2a295c8-1860-4315-ab35-764265d474e9","Type":"ContainerStarted","Data":"97b4ed4aeceec1b1001923847672a5a04d4ba5008e05b0b55a2809efa2a62132"}
Sep 09 20:33:43.204093 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.203577    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mv87t" event={"ID":"f2a295c8-1860-4315-ab35-764265d474e9","Type":"ContainerStarted","Data":"61e672b5ad13075908eeb7ea030597b2df3f9931c4822b1aeda690f37c0d8920"}
Sep 09 20:33:43.212976 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.212859    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nmmbf" event={"ID":"53ffa66d-d8a1-445f-a1cf-14d11c651095","Type":"ContainerStarted","Data":"ef21024e88f1cf27a0a0a02ded8643aa9a078182998e3def0f47ded74bc4cb67"}
Sep 09 20:33:43.219304 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.219208    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm" event={"ID":"c5ebe346-482a-410a-80b8-7b0a2981a38c","Type":"ContainerStarted","Data":"7316351cfacc5dfd9938a32072b74be1bc433202c135d2465736d4976317e69f"}
Sep 09 20:33:43.222448 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.222295    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cxbm6" event={"ID":"21002114-9907-455c-9324-a5ccb05b9516","Type":"ContainerStarted","Data":"40feb504eec4a811e4c217c5bef1fba73b968de66abbc5f7dd8b4627a18a7d87"}
Sep 09 20:33:43.662031 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.661996    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e972a506-9e4a-44dd-8924-1e56ad707ac6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.663104 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.663013    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e972a506-9e4a-44dd-8924-1e56ad707ac6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.833930 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.833882    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"]
Sep 09 20:33:43.836955 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.836934    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:43.840147 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.840114    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 09 20:33:43.840251 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.840191    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 09 20:33:43.840314 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.840120    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 09 20:33:43.840419 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.840407    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 09 20:33:43.840594 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.840568    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 09 20:33:43.841727 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.841293    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-hj7zl\""
Sep 09 20:33:43.841727 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.841296    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-573k7c1r8tmjp\""
Sep 09 20:33:43.850531 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.850504    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"]
Sep 09 20:33:43.855125 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.855099    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:33:43.964845 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.964422    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fa67c526-c03d-42ce-8c62-ccd903c21c5f-secret-thanos-querier-tls\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:43.964845 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.964474    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa67c526-c03d-42ce-8c62-ccd903c21c5f-metrics-client-ca\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:43.964845 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.964515    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fa67c526-c03d-42ce-8c62-ccd903c21c5f-secret-grpc-tls\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:43.964845 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.964555    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fa67c526-c03d-42ce-8c62-ccd903c21c5f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:43.964845 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.964588    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fa67c526-c03d-42ce-8c62-ccd903c21c5f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:43.964845 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.964625    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa67c526-c03d-42ce-8c62-ccd903c21c5f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:43.964845 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.964667    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-696b9\" (UniqueName: \"kubernetes.io/projected/fa67c526-c03d-42ce-8c62-ccd903c21c5f-kube-api-access-696b9\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:43.964845 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:43.964715    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa67c526-c03d-42ce-8c62-ccd903c21c5f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:44.065998 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:44.065950    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fa67c526-c03d-42ce-8c62-ccd903c21c5f-secret-grpc-tls\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:44.066170 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:44.066010    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fa67c526-c03d-42ce-8c62-ccd903c21c5f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:44.066170 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:44.066045    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fa67c526-c03d-42ce-8c62-ccd903c21c5f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:44.066170 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:44.066083    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa67c526-c03d-42ce-8c62-ccd903c21c5f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:44.066170 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:44.066131    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-696b9\" (UniqueName: \"kubernetes.io/projected/fa67c526-c03d-42ce-8c62-ccd903c21c5f-kube-api-access-696b9\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:44.066381 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:44.066180    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa67c526-c03d-42ce-8c62-ccd903c21c5f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:44.066381 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:44.066219    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fa67c526-c03d-42ce-8c62-ccd903c21c5f-secret-thanos-querier-tls\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:44.066381 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:44.066254    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa67c526-c03d-42ce-8c62-ccd903c21c5f-metrics-client-ca\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:44.068574 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:44.067034    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa67c526-c03d-42ce-8c62-ccd903c21c5f-metrics-client-ca\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:44.073768 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:44.073721    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fa67c526-c03d-42ce-8c62-ccd903c21c5f-secret-grpc-tls\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:44.075308 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:44.074839    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa67c526-c03d-42ce-8c62-ccd903c21c5f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:44.075308 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:44.075267    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fa67c526-c03d-42ce-8c62-ccd903c21c5f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:44.075854 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:44.075814    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fa67c526-c03d-42ce-8c62-ccd903c21c5f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:44.076766 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:44.076717    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa67c526-c03d-42ce-8c62-ccd903c21c5f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:44.077253 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:44.077210    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-696b9\" (UniqueName: \"kubernetes.io/projected/fa67c526-c03d-42ce-8c62-ccd903c21c5f-kube-api-access-696b9\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:44.078265 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:44.078224    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fa67c526-c03d-42ce-8c62-ccd903c21c5f-secret-thanos-querier-tls\") pod \"thanos-querier-55dddc5f75-fxg4r\" (UID: \"fa67c526-c03d-42ce-8c62-ccd903c21c5f\") " pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:44.148570 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:44.148495    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:44.233352 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:44.233264    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mv87t" event={"ID":"f2a295c8-1860-4315-ab35-764265d474e9","Type":"ContainerStarted","Data":"56d97bc2830c5887c3167fb3b9c5be39ad43286446e73748c882ffb357a52d72"}
Sep 09 20:33:44.237925 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:44.237164    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cxbm6" event={"ID":"21002114-9907-455c-9324-a5ccb05b9516","Type":"ContainerStarted","Data":"e55b6b14af39c6c9da767629184f0f10839e119ccbdd6584c076a26c75cc0a30"}
Sep 09 20:33:45.243093 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:45.243054    2556 generic.go:356] "Generic (PLEG): container finished" podID="21002114-9907-455c-9324-a5ccb05b9516" containerID="e55b6b14af39c6c9da767629184f0f10839e119ccbdd6584c076a26c75cc0a30" exitCode=0
Sep 09 20:33:45.243554 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:45.243135    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cxbm6" event={"ID":"21002114-9907-455c-9324-a5ccb05b9516","Type":"ContainerDied","Data":"e55b6b14af39c6c9da767629184f0f10839e119ccbdd6584c076a26c75cc0a30"}
Sep 09 20:33:46.130052 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.130010    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"]
Sep 09 20:33:46.165814 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.165753    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"]
Sep 09 20:33:46.165996 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.165880    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:33:46.169068 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.169035    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 20:33:46.169068 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.169065    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3mm5f1qidlke1\""
Sep 09 20:33:46.169274 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.169189    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 09 20:33:46.169274 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.169259    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 09 20:33:46.169372 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.169364    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9m2pd\""
Sep 09 20:33:46.169486 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.169469    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 09 20:33:46.283047 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.283016    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cbc0a8b7-eeab-44dc-9cbe-c3578953fd03-secret-metrics-server-client-certs\") pod \"metrics-server-5d64f9bfdb-v2gn4\" (UID: \"cbc0a8b7-eeab-44dc-9cbe-c3578953fd03\") " pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:33:46.283427 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.283058    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cbc0a8b7-eeab-44dc-9cbe-c3578953fd03-audit-log\") pod \"metrics-server-5d64f9bfdb-v2gn4\" (UID: \"cbc0a8b7-eeab-44dc-9cbe-c3578953fd03\") " pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:33:46.283427 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.283086    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cbc0a8b7-eeab-44dc-9cbe-c3578953fd03-client-ca-bundle\") pod \"metrics-server-5d64f9bfdb-v2gn4\" (UID: \"cbc0a8b7-eeab-44dc-9cbe-c3578953fd03\") " pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:33:46.283427 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.283160    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cbc0a8b7-eeab-44dc-9cbe-c3578953fd03-secret-metrics-server-tls\") pod \"metrics-server-5d64f9bfdb-v2gn4\" (UID: \"cbc0a8b7-eeab-44dc-9cbe-c3578953fd03\") " pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:33:46.283427 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.283196    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cbc0a8b7-eeab-44dc-9cbe-c3578953fd03-metrics-server-audit-profiles\") pod \"metrics-server-5d64f9bfdb-v2gn4\" (UID: \"cbc0a8b7-eeab-44dc-9cbe-c3578953fd03\") " pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:33:46.283427 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.283218    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ch6g\" (UniqueName: \"kubernetes.io/projected/cbc0a8b7-eeab-44dc-9cbe-c3578953fd03-kube-api-access-2ch6g\") pod \"metrics-server-5d64f9bfdb-v2gn4\" (UID: \"cbc0a8b7-eeab-44dc-9cbe-c3578953fd03\") " pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:33:46.283427 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.283327    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbc0a8b7-eeab-44dc-9cbe-c3578953fd03-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d64f9bfdb-v2gn4\" (UID: \"cbc0a8b7-eeab-44dc-9cbe-c3578953fd03\") " pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:33:46.384429 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.384339    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cbc0a8b7-eeab-44dc-9cbe-c3578953fd03-secret-metrics-server-client-certs\") pod \"metrics-server-5d64f9bfdb-v2gn4\" (UID: \"cbc0a8b7-eeab-44dc-9cbe-c3578953fd03\") " pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:33:46.384429 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.384394    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cbc0a8b7-eeab-44dc-9cbe-c3578953fd03-audit-log\") pod \"metrics-server-5d64f9bfdb-v2gn4\" (UID: \"cbc0a8b7-eeab-44dc-9cbe-c3578953fd03\") " pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:33:46.384429 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.384423    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cbc0a8b7-eeab-44dc-9cbe-c3578953fd03-client-ca-bundle\") pod \"metrics-server-5d64f9bfdb-v2gn4\" (UID: \"cbc0a8b7-eeab-44dc-9cbe-c3578953fd03\") " pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:33:46.384689 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.384475    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cbc0a8b7-eeab-44dc-9cbe-c3578953fd03-secret-metrics-server-tls\") pod \"metrics-server-5d64f9bfdb-v2gn4\" (UID: \"cbc0a8b7-eeab-44dc-9cbe-c3578953fd03\") " pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:33:46.384689 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.384504    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cbc0a8b7-eeab-44dc-9cbe-c3578953fd03-metrics-server-audit-profiles\") pod \"metrics-server-5d64f9bfdb-v2gn4\" (UID: \"cbc0a8b7-eeab-44dc-9cbe-c3578953fd03\") " pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:33:46.384689 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.384544    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ch6g\" (UniqueName: \"kubernetes.io/projected/cbc0a8b7-eeab-44dc-9cbe-c3578953fd03-kube-api-access-2ch6g\") pod \"metrics-server-5d64f9bfdb-v2gn4\" (UID: \"cbc0a8b7-eeab-44dc-9cbe-c3578953fd03\") " pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:33:46.384689 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.384597    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbc0a8b7-eeab-44dc-9cbe-c3578953fd03-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d64f9bfdb-v2gn4\" (UID: \"cbc0a8b7-eeab-44dc-9cbe-c3578953fd03\") " pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:33:46.384883 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.384838    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cbc0a8b7-eeab-44dc-9cbe-c3578953fd03-audit-log\") pod \"metrics-server-5d64f9bfdb-v2gn4\" (UID: \"cbc0a8b7-eeab-44dc-9cbe-c3578953fd03\") " pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:33:46.385406 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.385379    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbc0a8b7-eeab-44dc-9cbe-c3578953fd03-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d64f9bfdb-v2gn4\" (UID: \"cbc0a8b7-eeab-44dc-9cbe-c3578953fd03\") " pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:33:46.385658 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.385639    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cbc0a8b7-eeab-44dc-9cbe-c3578953fd03-metrics-server-audit-profiles\") pod \"metrics-server-5d64f9bfdb-v2gn4\" (UID: \"cbc0a8b7-eeab-44dc-9cbe-c3578953fd03\") " pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:33:46.387341 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.387305    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cbc0a8b7-eeab-44dc-9cbe-c3578953fd03-client-ca-bundle\") pod \"metrics-server-5d64f9bfdb-v2gn4\" (UID: \"cbc0a8b7-eeab-44dc-9cbe-c3578953fd03\") " pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:33:46.387485 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.387461    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cbc0a8b7-eeab-44dc-9cbe-c3578953fd03-secret-metrics-server-client-certs\") pod \"metrics-server-5d64f9bfdb-v2gn4\" (UID: \"cbc0a8b7-eeab-44dc-9cbe-c3578953fd03\") " pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:33:46.387717 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.387700    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cbc0a8b7-eeab-44dc-9cbe-c3578953fd03-secret-metrics-server-tls\") pod \"metrics-server-5d64f9bfdb-v2gn4\" (UID: \"cbc0a8b7-eeab-44dc-9cbe-c3578953fd03\") " pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:33:46.393238 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.393213    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ch6g\" (UniqueName: \"kubernetes.io/projected/cbc0a8b7-eeab-44dc-9cbe-c3578953fd03-kube-api-access-2ch6g\") pod \"metrics-server-5d64f9bfdb-v2gn4\" (UID: \"cbc0a8b7-eeab-44dc-9cbe-c3578953fd03\") " pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:33:46.477693 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.477652    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:33:46.605208 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.605177    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-tmchm"]
Sep 09 20:33:46.620218 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.620167    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-tmchm"]
Sep 09 20:33:46.620362 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.620272    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tmchm"
Sep 09 20:33:46.623083 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.623061    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 09 20:33:46.623210 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.623084    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-rkl7k\""
Sep 09 20:33:46.687407 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.687335    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e7a00cf1-7c14-49d7-9868-adc97c234e3f-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-tmchm\" (UID: \"e7a00cf1-7c14-49d7-9868-adc97c234e3f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tmchm"
Sep 09 20:33:46.687528 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.687489    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6zlb\" (UniqueName: \"kubernetes.io/projected/e7a00cf1-7c14-49d7-9868-adc97c234e3f-kube-api-access-x6zlb\") pod \"monitoring-plugin-85bbbbbf4-tmchm\" (UID: \"e7a00cf1-7c14-49d7-9868-adc97c234e3f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tmchm"
Sep 09 20:33:46.788899 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.788866    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e7a00cf1-7c14-49d7-9868-adc97c234e3f-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-tmchm\" (UID: \"e7a00cf1-7c14-49d7-9868-adc97c234e3f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tmchm"
Sep 09 20:33:46.789065 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.788944    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6zlb\" (UniqueName: \"kubernetes.io/projected/e7a00cf1-7c14-49d7-9868-adc97c234e3f-kube-api-access-x6zlb\") pod \"monitoring-plugin-85bbbbbf4-tmchm\" (UID: \"e7a00cf1-7c14-49d7-9868-adc97c234e3f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tmchm"
Sep 09 20:33:46.791460 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.791428    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e7a00cf1-7c14-49d7-9868-adc97c234e3f-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-tmchm\" (UID: \"e7a00cf1-7c14-49d7-9868-adc97c234e3f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tmchm"
Sep 09 20:33:46.801800 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.801781    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6zlb\" (UniqueName: \"kubernetes.io/projected/e7a00cf1-7c14-49d7-9868-adc97c234e3f-kube-api-access-x6zlb\") pod \"monitoring-plugin-85bbbbbf4-tmchm\" (UID: \"e7a00cf1-7c14-49d7-9868-adc97c234e3f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tmchm"
Sep 09 20:33:46.933891 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:46.933866    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tmchm"
Sep 09 20:33:47.749210 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:47.748190    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:33:47.749210 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:47.748998    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"]
Sep 09 20:33:47.750763 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:33:47.750722    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode972a506_9e4a_44dd_8924_1e56ad707ac6.slice/crio-e8a9434bc0bfb5d1eb4a9ca74a7925ee369690e199385d55344b6eb6d0dbb89d WatchSource:0}: Error finding container e8a9434bc0bfb5d1eb4a9ca74a7925ee369690e199385d55344b6eb6d0dbb89d: Status 404 returned error can't find the container with id e8a9434bc0bfb5d1eb4a9ca74a7925ee369690e199385d55344b6eb6d0dbb89d
Sep 09 20:33:47.752735 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:33:47.751781    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc0a8b7_eeab_44dc_9cbe_c3578953fd03.slice/crio-c4b3b9ec3ff57da91d0fbb156cafab01cf819d0fb5dbdc7ae2374f5a02df832e WatchSource:0}: Error finding container c4b3b9ec3ff57da91d0fbb156cafab01cf819d0fb5dbdc7ae2374f5a02df832e: Status 404 returned error can't find the container with id c4b3b9ec3ff57da91d0fbb156cafab01cf819d0fb5dbdc7ae2374f5a02df832e
Sep 09 20:33:47.757430 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:47.756929    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"]
Sep 09 20:33:47.758429 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:47.758321    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-tmchm"]
Sep 09 20:33:47.761393 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:33:47.761364    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa67c526_c03d_42ce_8c62_ccd903c21c5f.slice/crio-6d91f34e038188b139ce4f38c1a7e8363f09a0185543aec864eb8092c09219ba WatchSource:0}: Error finding container 6d91f34e038188b139ce4f38c1a7e8363f09a0185543aec864eb8092c09219ba: Status 404 returned error can't find the container with id 6d91f34e038188b139ce4f38c1a7e8363f09a0185543aec864eb8092c09219ba
Sep 09 20:33:47.764210 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:33:47.764169    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a00cf1_7c14_49d7_9868_adc97c234e3f.slice/crio-d94570db1e3eda67e8308309cfc44d7210b04e195151e2b45fc47030a119cf52 WatchSource:0}: Error finding container d94570db1e3eda67e8308309cfc44d7210b04e195151e2b45fc47030a119cf52: Status 404 returned error can't find the container with id d94570db1e3eda67e8308309cfc44d7210b04e195151e2b45fc47030a119cf52
Sep 09 20:33:48.075293 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.075252    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:33:48.080032 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.080007    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.083285 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.083101    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 20:33:48.084966 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.084619    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lhhrg\""
Sep 09 20:33:48.084966 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.084808    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 20:33:48.085228 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.085200    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 20:33:48.085323 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.085254    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 20:33:48.085323 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.085291    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 20:33:48.085413 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.085368    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 20:33:48.085531 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.085510    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 20:33:48.085531 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.085517    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 20:33:48.085682 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.085578    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 20:33:48.086012 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.085945    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 20:33:48.086012 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.085956    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 20:33:48.086282 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.086258    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 20:33:48.086511 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.086492    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3fe50p7bjlk4k\""
Sep 09 20:33:48.088791 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.088743    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 20:33:48.091896 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.091879    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 20:33:48.096733 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.096655    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:33:48.201852 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.201817    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.201852 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.201854    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh42r\" (UniqueName: \"kubernetes.io/projected/06acd67c-1f5b-4053-9aa9-5147c56411e4-kube-api-access-jh42r\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.202078 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.201877    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-config\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.202078 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.201932    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.202078 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.201965    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.202078 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.201981    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.202078 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.202005    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-web-config\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.202078 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.202044    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.202078 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.202064    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.202435 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.202086    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.202435 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.202136    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.202435 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.202197    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.202435 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.202226    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06acd67c-1f5b-4053-9aa9-5147c56411e4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.202435 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.202253    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06acd67c-1f5b-4053-9aa9-5147c56411e4-config-out\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.202435 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.202281    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.202435 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.202306    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.202435 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.202336    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.202435 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.202362    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.202435 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.202394    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.202435 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.202416    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.257946 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.257889    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mv87t" event={"ID":"f2a295c8-1860-4315-ab35-764265d474e9","Type":"ContainerStarted","Data":"f9016387618f6ee247479d3fdd152bc5314023a89aa61806d639a53edbfff767"}
Sep 09 20:33:48.259093 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.259069    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tmchm" event={"ID":"e7a00cf1-7c14-49d7-9868-adc97c234e3f","Type":"ContainerStarted","Data":"d94570db1e3eda67e8308309cfc44d7210b04e195151e2b45fc47030a119cf52"}
Sep 09 20:33:48.263223 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.263192    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q2zw5" event={"ID":"61bb5c63-cbb7-4186-9c23-5706dba8cec2","Type":"ContainerStarted","Data":"eea3b397c5f91116409f1baa238925ac54f111717efb4a44a9cfb2c27420afbc"}
Sep 09 20:33:48.266803 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.266779    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nmmbf" event={"ID":"53ffa66d-d8a1-445f-a1cf-14d11c651095","Type":"ContainerStarted","Data":"04b0894a5f6607d76c93538bc857c8d15aedb8480f88e9feb03c3efdd5f6afe0"}
Sep 09 20:33:48.268551 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.268524    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2zpmx" event={"ID":"14c3f4cb-c792-4945-bd65-932ed6a27aa9","Type":"ContainerStarted","Data":"97ceedfb6af330960f83029ce6d040aab084ae5d929778284048d073ba3717e5"}
Sep 09 20:33:48.268651 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.268587    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:33:48.271125 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.271093    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm" event={"ID":"c5ebe346-482a-410a-80b8-7b0a2981a38c","Type":"ContainerStarted","Data":"e1251f49782647b3c7596e7236e9cf86fabe72f5cf1028336a9c1f5bdac406e4"}
Sep 09 20:33:48.271233 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.271136    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm" event={"ID":"c5ebe346-482a-410a-80b8-7b0a2981a38c","Type":"ContainerStarted","Data":"31b6a3c1f3bea7a39a4833fffd4e453352540d4f07e78fad861f3e34e6f37c3c"}
Sep 09 20:33:48.271233 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.271170    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm" event={"ID":"c5ebe346-482a-410a-80b8-7b0a2981a38c","Type":"ContainerStarted","Data":"c2b6627fd36bef4ba6bf26e8562b5c269da92d9fafca08e7fc438e43fcac1482"}
Sep 09 20:33:48.273447 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.273424    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cxbm6" event={"ID":"21002114-9907-455c-9324-a5ccb05b9516","Type":"ContainerStarted","Data":"eaf5eaa04ba0254cafe8963d2edc2daad589dbf35f26856e7669deca35248fac"}
Sep 09 20:33:48.273563 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.273469    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cxbm6" event={"ID":"21002114-9907-455c-9324-a5ccb05b9516","Type":"ContainerStarted","Data":"6d824803149ddab88e888de1192bf909058715f0050bd7683c34214ac4935ffb"}
Sep 09 20:33:48.274889 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.274869    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4" event={"ID":"cbc0a8b7-eeab-44dc-9cbe-c3578953fd03","Type":"ContainerStarted","Data":"c4b3b9ec3ff57da91d0fbb156cafab01cf819d0fb5dbdc7ae2374f5a02df832e"}
Sep 09 20:33:48.276012 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.275994    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r" event={"ID":"fa67c526-c03d-42ce-8c62-ccd903c21c5f","Type":"ContainerStarted","Data":"6d91f34e038188b139ce4f38c1a7e8363f09a0185543aec864eb8092c09219ba"}
Sep 09 20:33:48.277303 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.277285    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e972a506-9e4a-44dd-8924-1e56ad707ac6","Type":"ContainerStarted","Data":"e8a9434bc0bfb5d1eb4a9ca74a7925ee369690e199385d55344b6eb6d0dbb89d"}
Sep 09 20:33:48.298980 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.298870    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mv87t" podStartSLOduration=3.006734141 podStartE2EDuration="7.298851118s" totalImagesPullingTime="4.292116977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:48.293994034 +0000 UTC m=+73.973954344" watchObservedRunningTime="2026-09-09 20:33:48.298851118 +0000 UTC m=+73.978811420"
Sep 09 20:33:48.303573 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.303549    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh42r\" (UniqueName: \"kubernetes.io/projected/06acd67c-1f5b-4053-9aa9-5147c56411e4-kube-api-access-jh42r\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.303663 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.303587    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-config\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.303663 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.303609    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.304037 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.303712    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.304135 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.304068    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.304135 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.304096    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-web-config\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.304135 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.304132    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.304306 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.304160    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.304306 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.304176    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.304306 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.304231    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.304306 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.304273    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.304306 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.304299    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06acd67c-1f5b-4053-9aa9-5147c56411e4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.304629 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.304323    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06acd67c-1f5b-4053-9aa9-5147c56411e4-config-out\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.304629 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.304349    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.304629 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.304374    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.304629 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.304404    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.304629 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.304429    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.304629 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.304466    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.304629 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.304490    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.304629 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.304558    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.305047 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.304814    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.306734 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.305751    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.306734 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.306286    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.306734 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.306304    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.306734 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.306623    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.306734 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.306672    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-config\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.307056 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.306748    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.307388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.307364    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.307471 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.307445    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.308049 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.308029    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.309084 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.309054    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.309348 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.309307    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06acd67c-1f5b-4053-9aa9-5147c56411e4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.309448 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.309428    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06acd67c-1f5b-4053-9aa9-5147c56411e4-config-out\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.309859 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.309835    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.310091 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.310072    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.310418 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.310385    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.310701 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.310662    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-web-config\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.311872 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.311844    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.311973 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.311932    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.317013 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.316974    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh42r\" (UniqueName: \"kubernetes.io/projected/06acd67c-1f5b-4053-9aa9-5147c56411e4-kube-api-access-jh42r\") pod \"prometheus-k8s-0\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.323759 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.323715    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cxbm6" podStartSLOduration=5.842431423 podStartE2EDuration="7.323701697s" totalImagesPullingTime="1.402923943s" totalInitContainerRuntime="78.346331ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:48.323405003 +0000 UTC m=+74.003365341" watchObservedRunningTime="2026-09-09 20:33:48.323701697 +0000 UTC m=+74.003662000"
Sep 09 20:33:48.350956 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.350873    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2zpmx" podStartSLOduration=68.009239302 podStartE2EDuration="1m14.350859066s" totalImagesPullingTime="6.341619764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:32:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:48.3495368 +0000 UTC m=+74.029497110" watchObservedRunningTime="2026-09-09 20:33:48.350859066 +0000 UTC m=+74.030819374"
Sep 09 20:33:48.378267 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.378220    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-4fwvm" podStartSLOduration=3.022270142 podStartE2EDuration="7.378184056s" totalImagesPullingTime="4.355913914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:48.377053257 +0000 UTC m=+74.057013581" watchObservedRunningTime="2026-09-09 20:33:48.378184056 +0000 UTC m=+74.058144383"
Sep 09 20:33:48.393779 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.393713    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:33:48.394218 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.394178    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-q2zw5" podStartSLOduration=67.048814731 podStartE2EDuration="1m13.394163949s" totalImagesPullingTime="6.345349218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:32:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:48.393145945 +0000 UTC m=+74.073106255" watchObservedRunningTime="2026-09-09 20:33:48.394163949 +0000 UTC m=+74.074124258"
Sep 09 20:33:48.420971 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.420930    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nmmbf" podStartSLOduration=34.0457936 podStartE2EDuration="39.420898655s" totalImagesPullingTime="5.375105055s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:33:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:48.419547719 +0000 UTC m=+74.099508033" watchObservedRunningTime="2026-09-09 20:33:48.420898655 +0000 UTC m=+74.100859004"
Sep 09 20:33:48.688159 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:48.687789    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:33:48.689887 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:33:48.689856    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06acd67c_1f5b_4053_9aa9_5147c56411e4.slice/crio-3d8bcc85e0ada2cf47fc182d05ba4a17ba1a72dcec379c90daafbe570ee7df9a WatchSource:0}: Error finding container 3d8bcc85e0ada2cf47fc182d05ba4a17ba1a72dcec379c90daafbe570ee7df9a: Status 404 returned error can't find the container with id 3d8bcc85e0ada2cf47fc182d05ba4a17ba1a72dcec379c90daafbe570ee7df9a
Sep 09 20:33:49.289630 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:49.289582    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06acd67c-1f5b-4053-9aa9-5147c56411e4","Type":"ContainerStarted","Data":"3d8bcc85e0ada2cf47fc182d05ba4a17ba1a72dcec379c90daafbe570ee7df9a"}
Sep 09 20:33:51.299937 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:51.299890    2556 generic.go:356] "Generic (PLEG): container finished" podID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerID="634e954d3f23d811e12adb999919356ae5cc43d619a2d8d3948e3764f3a83ae9" exitCode=0
Sep 09 20:33:51.300336 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:51.299978    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06acd67c-1f5b-4053-9aa9-5147c56411e4","Type":"ContainerDied","Data":"634e954d3f23d811e12adb999919356ae5cc43d619a2d8d3948e3764f3a83ae9"}
Sep 09 20:33:51.302361 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:51.302342    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r" event={"ID":"fa67c526-c03d-42ce-8c62-ccd903c21c5f","Type":"ContainerStarted","Data":"c13ba784337ce02a0bf731cfa593911e4d84e1bd83ae539afb164bdfea461bd9"}
Sep 09 20:33:51.302447 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:51.302398    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r" event={"ID":"fa67c526-c03d-42ce-8c62-ccd903c21c5f","Type":"ContainerStarted","Data":"7f6a6a4cb6c0159f7172c61565cf52442a40122b7ef412644d3223e167cc0f67"}
Sep 09 20:33:51.302447 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:51.302426    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r" event={"ID":"fa67c526-c03d-42ce-8c62-ccd903c21c5f","Type":"ContainerStarted","Data":"a0ab570d7a54a842df54e1fa1d4ede6e86b4eccaf775ca646a209252902cddd3"}
Sep 09 20:33:51.303860 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:51.303841    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tmchm" event={"ID":"e7a00cf1-7c14-49d7-9868-adc97c234e3f","Type":"ContainerStarted","Data":"b05ccf3182e1427b84c3698574be78476be70257e8ac0c4944b1d0d6bc838ec5"}
Sep 09 20:33:51.304079 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:51.304057    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tmchm"
Sep 09 20:33:51.308933 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:51.306074    2556 generic.go:356] "Generic (PLEG): container finished" podID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerID="c751b7ce5b900822f61e4e7c7b1a16fe198f60ad58b378db61034e0f6e3c1867" exitCode=0
Sep 09 20:33:51.308933 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:51.306231    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e972a506-9e4a-44dd-8924-1e56ad707ac6","Type":"ContainerDied","Data":"c751b7ce5b900822f61e4e7c7b1a16fe198f60ad58b378db61034e0f6e3c1867"}
Sep 09 20:33:51.312939 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:51.312896    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4" event={"ID":"cbc0a8b7-eeab-44dc-9cbe-c3578953fd03","Type":"ContainerStarted","Data":"bc1cc8b9f43a71c7b9ea731bca2129ed4a2553cfe281687177193abc3dc082e0"}
Sep 09 20:33:51.314460 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:51.314443    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tmchm"
Sep 09 20:33:51.349916 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:51.349860    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4" podStartSLOduration=2.261946804 podStartE2EDuration="5.349843145s" totalImagesPullingTime="3.087896341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:51.348224069 +0000 UTC m=+77.028184381" watchObservedRunningTime="2026-09-09 20:33:51.349843145 +0000 UTC m=+77.029803455"
Sep 09 20:33:51.385154 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:51.385072    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tmchm" podStartSLOduration=2.30493544 podStartE2EDuration="5.385056896s" totalImagesPullingTime="3.080121456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:51.38487372 +0000 UTC m=+77.064834032" watchObservedRunningTime="2026-09-09 20:33:51.385056896 +0000 UTC m=+77.065017210"
Sep 09 20:33:52.148073 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:52.148052    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-676b97c989-qctgc"
Sep 09 20:33:52.323291 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:52.323237    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r" event={"ID":"fa67c526-c03d-42ce-8c62-ccd903c21c5f","Type":"ContainerStarted","Data":"9d06c39b2c1acd4b173390ec2aca0ddcd4d412579fff3350ce7a5d9a264dddd5"}
Sep 09 20:33:52.323707 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:52.323306    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r" event={"ID":"fa67c526-c03d-42ce-8c62-ccd903c21c5f","Type":"ContainerStarted","Data":"d6d0f63ff00350aa56d8db598f64a4f9014e703a55a2d356638c79e0aeeecbb6"}
Sep 09 20:33:52.323707 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:52.323333    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r" event={"ID":"fa67c526-c03d-42ce-8c62-ccd903c21c5f","Type":"ContainerStarted","Data":"b2d9cc6c4d3133b849f9b30f02962125144be53a46e20160985bfae7ad29c240"}
Sep 09 20:33:52.323838 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:52.323805    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:52.348147 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:52.348092    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r" podStartSLOduration=5.417440162 podStartE2EDuration="9.348071643s" totalImagesPullingTime="3.930631481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:52.345526676 +0000 UTC m=+78.025486986" watchObservedRunningTime="2026-09-09 20:33:52.348071643 +0000 UTC m=+78.028031954"
Sep 09 20:33:54.336466 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:54.336427    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e972a506-9e4a-44dd-8924-1e56ad707ac6","Type":"ContainerStarted","Data":"a0ba094c334cdd4f5f061a05fa13c216d986c5f2fcb8c4cb22a7b3f0ef8ad2c2"}
Sep 09 20:33:54.336866 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:54.336560    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e972a506-9e4a-44dd-8924-1e56ad707ac6","Type":"ContainerStarted","Data":"9194faf246cb5dfc0fde515b827cbd853ae8db73c167442cb973dc3656e5cdac"}
Sep 09 20:33:55.344600 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:55.344568    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e972a506-9e4a-44dd-8924-1e56ad707ac6","Type":"ContainerStarted","Data":"85484e2e9e951ca6cb073b29cf3b3dff41f4bbb22997db2bc3f2ff96925162e4"}
Sep 09 20:33:55.344987 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:55.344630    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e972a506-9e4a-44dd-8924-1e56ad707ac6","Type":"ContainerStarted","Data":"f0543141fc2af09cacddc16db26fda4e4751c57506860b4fce1f9f33b791d637"}
Sep 09 20:33:55.344987 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:55.344673    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e972a506-9e4a-44dd-8924-1e56ad707ac6","Type":"ContainerStarted","Data":"7770ed6b37ea9bb43dcafbde34818dd77d62993f0dbf9abe3c2e6b163af2879d"}
Sep 09 20:33:55.344987 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:55.344706    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e972a506-9e4a-44dd-8924-1e56ad707ac6","Type":"ContainerStarted","Data":"791dd8f32c09dbc4776f552d9ff20d420675a21e27cb2b2162c8118ce869685a"}
Sep 09 20:33:55.347865 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:55.347842    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06acd67c-1f5b-4053-9aa9-5147c56411e4","Type":"ContainerStarted","Data":"b4cbaff0dcf0d205880edb2a2301cd3a399b5767dbc952d6a3713058ce82364d"}
Sep 09 20:33:55.347992 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:55.347892    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06acd67c-1f5b-4053-9aa9-5147c56411e4","Type":"ContainerStarted","Data":"fbd0aac93c2e44484ed9092c7c421b15f229218bae4c1b45045656736054ab06"}
Sep 09 20:33:55.347992 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:55.347941    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06acd67c-1f5b-4053-9aa9-5147c56411e4","Type":"ContainerStarted","Data":"a8b25005a7a0eea4db6f01967dd87ec29d904aeb405b16663d2ad8e692c93f63"}
Sep 09 20:33:55.347992 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:55.347967    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06acd67c-1f5b-4053-9aa9-5147c56411e4","Type":"ContainerStarted","Data":"528a4219cf514afb314003c0e2d89d0992be6d850752c7f95df502bce963de03"}
Sep 09 20:33:55.348095 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:55.347994    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06acd67c-1f5b-4053-9aa9-5147c56411e4","Type":"ContainerStarted","Data":"a7259e847e92e1956caa72606b12bcbb1fec2162c9793d2ccc9520d153a44b20"}
Sep 09 20:33:55.348095 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:55.348016    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06acd67c-1f5b-4053-9aa9-5147c56411e4","Type":"ContainerStarted","Data":"992ad2e864b99ebef689e51d5732730c32c1a091c44826cadec41bde4878b0eb"}
Sep 09 20:33:55.370524 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:55.370477    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=8.243176138 podStartE2EDuration="13.370439827s" totalImagesPullingTime="5.105324411s" totalInitContainerRuntime="21.939278ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:55.369083965 +0000 UTC m=+81.049044277" watchObservedRunningTime="2026-09-09 20:33:55.370439827 +0000 UTC m=+81.050400129"
Sep 09 20:33:55.397029 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:55.396981    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.9738430569999998 podStartE2EDuration="7.39696665s" totalImagesPullingTime="5.401545645s" totalInitContainerRuntime="21.577948ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:55.394874004 +0000 UTC m=+81.074834326" watchObservedRunningTime="2026-09-09 20:33:55.39696665 +0000 UTC m=+81.076926958"
Sep 09 20:33:58.333840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:58.333737    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-55dddc5f75-fxg4r"
Sep 09 20:33:58.394170 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:33:58.394149    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:34:06.478135 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:34:06.478100    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:34:06.478610 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:34:06.478161    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:34:11.825343 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:34:11.825314    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vtzkg_420e8660-43a1-4d78-841b-75448e999ac8/serve-healthcheck-canary/0.log"
Sep 09 20:34:19.293878 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:34:19.293848    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2zpmx"
Sep 09 20:34:26.487634 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:34:26.487605    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:34:26.495260 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:34:26.495229    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5d64f9bfdb-v2gn4"
Sep 09 20:34:48.394543 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:34:48.394504    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:34:48.413456 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:34:48.413434    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:34:48.576685 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:34:48.576657    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:02.123852 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:02.123813    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:35:02.124346 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:02.124209    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="alertmanager" containerID="cri-o://9194faf246cb5dfc0fde515b827cbd853ae8db73c167442cb973dc3656e5cdac" gracePeriod=120
Sep 09 20:35:02.124346 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:02.124283    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="kube-rbac-proxy-metric" containerID="cri-o://f0543141fc2af09cacddc16db26fda4e4751c57506860b4fce1f9f33b791d637" gracePeriod=120
Sep 09 20:35:02.124346 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:02.124335    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="kube-rbac-proxy" containerID="cri-o://7770ed6b37ea9bb43dcafbde34818dd77d62993f0dbf9abe3c2e6b163af2879d" gracePeriod=120
Sep 09 20:35:02.124515 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:02.124290    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="kube-rbac-proxy-web" containerID="cri-o://791dd8f32c09dbc4776f552d9ff20d420675a21e27cb2b2162c8118ce869685a" gracePeriod=120
Sep 09 20:35:02.124515 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:02.124415    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="config-reloader" containerID="cri-o://a0ba094c334cdd4f5f061a05fa13c216d986c5f2fcb8c4cb22a7b3f0ef8ad2c2" gracePeriod=120
Sep 09 20:35:02.124515 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:02.124477    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="prom-label-proxy" containerID="cri-o://85484e2e9e951ca6cb073b29cf3b3dff41f4bbb22997db2bc3f2ff96925162e4" gracePeriod=120
Sep 09 20:35:02.619233 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:02.619203    2556 generic.go:356] "Generic (PLEG): container finished" podID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerID="85484e2e9e951ca6cb073b29cf3b3dff41f4bbb22997db2bc3f2ff96925162e4" exitCode=0
Sep 09 20:35:02.619233 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:02.619229    2556 generic.go:356] "Generic (PLEG): container finished" podID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerID="7770ed6b37ea9bb43dcafbde34818dd77d62993f0dbf9abe3c2e6b163af2879d" exitCode=0
Sep 09 20:35:02.619233 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:02.619236    2556 generic.go:356] "Generic (PLEG): container finished" podID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerID="a0ba094c334cdd4f5f061a05fa13c216d986c5f2fcb8c4cb22a7b3f0ef8ad2c2" exitCode=0
Sep 09 20:35:02.619429 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:02.619244    2556 generic.go:356] "Generic (PLEG): container finished" podID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerID="9194faf246cb5dfc0fde515b827cbd853ae8db73c167442cb973dc3656e5cdac" exitCode=0
Sep 09 20:35:02.619429 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:02.619266    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e972a506-9e4a-44dd-8924-1e56ad707ac6","Type":"ContainerDied","Data":"85484e2e9e951ca6cb073b29cf3b3dff41f4bbb22997db2bc3f2ff96925162e4"}
Sep 09 20:35:02.619429 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:02.619318    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e972a506-9e4a-44dd-8924-1e56ad707ac6","Type":"ContainerDied","Data":"7770ed6b37ea9bb43dcafbde34818dd77d62993f0dbf9abe3c2e6b163af2879d"}
Sep 09 20:35:02.619429 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:02.619352    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e972a506-9e4a-44dd-8924-1e56ad707ac6","Type":"ContainerDied","Data":"a0ba094c334cdd4f5f061a05fa13c216d986c5f2fcb8c4cb22a7b3f0ef8ad2c2"}
Sep 09 20:35:02.619429 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:02.619377    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e972a506-9e4a-44dd-8924-1e56ad707ac6","Type":"ContainerDied","Data":"9194faf246cb5dfc0fde515b827cbd853ae8db73c167442cb973dc3656e5cdac"}
Sep 09 20:35:03.364639 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.364617    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.484388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.484366    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e972a506-9e4a-44dd-8924-1e56ad707ac6-alertmanager-trusted-ca-bundle\") pod \"e972a506-9e4a-44dd-8924-1e56ad707ac6\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") "
Sep 09 20:35:03.484513 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.484420    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"e972a506-9e4a-44dd-8924-1e56ad707ac6\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") "
Sep 09 20:35:03.484513 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.484451    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-main-tls\") pod \"e972a506-9e4a-44dd-8924-1e56ad707ac6\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") "
Sep 09 20:35:03.484513 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.484471    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-kube-rbac-proxy\") pod \"e972a506-9e4a-44dd-8924-1e56ad707ac6\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") "
Sep 09 20:35:03.484513 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.484492    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zdk9\" (UniqueName: \"kubernetes.io/projected/e972a506-9e4a-44dd-8924-1e56ad707ac6-kube-api-access-5zdk9\") pod \"e972a506-9e4a-44dd-8924-1e56ad707ac6\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") "
Sep 09 20:35:03.484735 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.484527    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-config-volume\") pod \"e972a506-9e4a-44dd-8924-1e56ad707ac6\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") "
Sep 09 20:35:03.484735 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.484556    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e972a506-9e4a-44dd-8924-1e56ad707ac6-metrics-client-ca\") pod \"e972a506-9e4a-44dd-8924-1e56ad707ac6\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") "
Sep 09 20:35:03.484735 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.484604    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-cluster-tls-config\") pod \"e972a506-9e4a-44dd-8924-1e56ad707ac6\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") "
Sep 09 20:35:03.484735 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.484637    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-web-config\") pod \"e972a506-9e4a-44dd-8924-1e56ad707ac6\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") "
Sep 09 20:35:03.484735 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.484659    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e972a506-9e4a-44dd-8924-1e56ad707ac6-config-out\") pod \"e972a506-9e4a-44dd-8924-1e56ad707ac6\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") "
Sep 09 20:35:03.484735 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.484689    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-kube-rbac-proxy-web\") pod \"e972a506-9e4a-44dd-8924-1e56ad707ac6\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") "
Sep 09 20:35:03.484735 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.484721    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e972a506-9e4a-44dd-8924-1e56ad707ac6-tls-assets\") pod \"e972a506-9e4a-44dd-8924-1e56ad707ac6\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") "
Sep 09 20:35:03.485113 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.484751    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e972a506-9e4a-44dd-8924-1e56ad707ac6-alertmanager-main-db\") pod \"e972a506-9e4a-44dd-8924-1e56ad707ac6\" (UID: \"e972a506-9e4a-44dd-8924-1e56ad707ac6\") "
Sep 09 20:35:03.485113 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.484765    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e972a506-9e4a-44dd-8924-1e56ad707ac6-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "e972a506-9e4a-44dd-8924-1e56ad707ac6" (UID: "e972a506-9e4a-44dd-8924-1e56ad707ac6"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:03.485113 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.485069    2556 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e972a506-9e4a-44dd-8924-1e56ad707ac6-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:03.485370 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.485344    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e972a506-9e4a-44dd-8924-1e56ad707ac6-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "e972a506-9e4a-44dd-8924-1e56ad707ac6" (UID: "e972a506-9e4a-44dd-8924-1e56ad707ac6"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:03.485616 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.485591    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e972a506-9e4a-44dd-8924-1e56ad707ac6-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "e972a506-9e4a-44dd-8924-1e56ad707ac6" (UID: "e972a506-9e4a-44dd-8924-1e56ad707ac6"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:03.487594 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.487562    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e972a506-9e4a-44dd-8924-1e56ad707ac6-kube-api-access-5zdk9" (OuterVolumeSpecName: "kube-api-access-5zdk9") pod "e972a506-9e4a-44dd-8924-1e56ad707ac6" (UID: "e972a506-9e4a-44dd-8924-1e56ad707ac6"). InnerVolumeSpecName "kube-api-access-5zdk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:03.488065 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.487859    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-config-volume" (OuterVolumeSpecName: "config-volume") pod "e972a506-9e4a-44dd-8924-1e56ad707ac6" (UID: "e972a506-9e4a-44dd-8924-1e56ad707ac6"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:03.488065 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.488046    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "e972a506-9e4a-44dd-8924-1e56ad707ac6" (UID: "e972a506-9e4a-44dd-8924-1e56ad707ac6"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:03.488338 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.488303    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "e972a506-9e4a-44dd-8924-1e56ad707ac6" (UID: "e972a506-9e4a-44dd-8924-1e56ad707ac6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:03.488445 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.488414    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "e972a506-9e4a-44dd-8924-1e56ad707ac6" (UID: "e972a506-9e4a-44dd-8924-1e56ad707ac6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:03.488587 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.488562    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "e972a506-9e4a-44dd-8924-1e56ad707ac6" (UID: "e972a506-9e4a-44dd-8924-1e56ad707ac6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:03.488769 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.488747    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e972a506-9e4a-44dd-8924-1e56ad707ac6-config-out" (OuterVolumeSpecName: "config-out") pod "e972a506-9e4a-44dd-8924-1e56ad707ac6" (UID: "e972a506-9e4a-44dd-8924-1e56ad707ac6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:03.489046 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.489028    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e972a506-9e4a-44dd-8924-1e56ad707ac6-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e972a506-9e4a-44dd-8924-1e56ad707ac6" (UID: "e972a506-9e4a-44dd-8924-1e56ad707ac6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:03.492863 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.492837    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "e972a506-9e4a-44dd-8924-1e56ad707ac6" (UID: "e972a506-9e4a-44dd-8924-1e56ad707ac6"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:03.499279 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.499259    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-web-config" (OuterVolumeSpecName: "web-config") pod "e972a506-9e4a-44dd-8924-1e56ad707ac6" (UID: "e972a506-9e4a-44dd-8924-1e56ad707ac6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:03.585693 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.585657    2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-main-tls\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:03.585693 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.585694    2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:03.585849 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.585709    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zdk9\" (UniqueName: \"kubernetes.io/projected/e972a506-9e4a-44dd-8924-1e56ad707ac6-kube-api-access-5zdk9\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:03.585849 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.585722    2556 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-config-volume\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:03.585849 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.585734    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e972a506-9e4a-44dd-8924-1e56ad707ac6-metrics-client-ca\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:03.585849 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.585746    2556 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-cluster-tls-config\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:03.585849 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.585758    2556 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-web-config\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:03.585849 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.585771    2556 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e972a506-9e4a-44dd-8924-1e56ad707ac6-config-out\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:03.585849 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.585783    2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:03.585849 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.585795    2556 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e972a506-9e4a-44dd-8924-1e56ad707ac6-tls-assets\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:03.585849 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.585809    2556 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e972a506-9e4a-44dd-8924-1e56ad707ac6-alertmanager-main-db\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:03.585849 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.585823    2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e972a506-9e4a-44dd-8924-1e56ad707ac6-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:03.626746 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.626722    2556 generic.go:356] "Generic (PLEG): container finished" podID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerID="f0543141fc2af09cacddc16db26fda4e4751c57506860b4fce1f9f33b791d637" exitCode=0
Sep 09 20:35:03.626746 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.626744    2556 generic.go:356] "Generic (PLEG): container finished" podID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerID="791dd8f32c09dbc4776f552d9ff20d420675a21e27cb2b2162c8118ce869685a" exitCode=0
Sep 09 20:35:03.626926 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.626796    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.626926 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.626805    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e972a506-9e4a-44dd-8924-1e56ad707ac6","Type":"ContainerDied","Data":"f0543141fc2af09cacddc16db26fda4e4751c57506860b4fce1f9f33b791d637"}
Sep 09 20:35:03.626926 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.626893    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e972a506-9e4a-44dd-8924-1e56ad707ac6","Type":"ContainerDied","Data":"791dd8f32c09dbc4776f552d9ff20d420675a21e27cb2b2162c8118ce869685a"}
Sep 09 20:35:03.627067 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.626949    2556 scope.go:117] "RemoveContainer" containerID="85484e2e9e951ca6cb073b29cf3b3dff41f4bbb22997db2bc3f2ff96925162e4"
Sep 09 20:35:03.627067 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.626963    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e972a506-9e4a-44dd-8924-1e56ad707ac6","Type":"ContainerDied","Data":"e8a9434bc0bfb5d1eb4a9ca74a7925ee369690e199385d55344b6eb6d0dbb89d"}
Sep 09 20:35:03.634534 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.634517    2556 scope.go:117] "RemoveContainer" containerID="f0543141fc2af09cacddc16db26fda4e4751c57506860b4fce1f9f33b791d637"
Sep 09 20:35:03.641284 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.641265    2556 scope.go:117] "RemoveContainer" containerID="7770ed6b37ea9bb43dcafbde34818dd77d62993f0dbf9abe3c2e6b163af2879d"
Sep 09 20:35:03.648469 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.648448    2556 scope.go:117] "RemoveContainer" containerID="791dd8f32c09dbc4776f552d9ff20d420675a21e27cb2b2162c8118ce869685a"
Sep 09 20:35:03.651402 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.651381    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:35:03.655286 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.655266    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:35:03.656803 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.656790    2556 scope.go:117] "RemoveContainer" containerID="a0ba094c334cdd4f5f061a05fa13c216d986c5f2fcb8c4cb22a7b3f0ef8ad2c2"
Sep 09 20:35:03.663350 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.663333    2556 scope.go:117] "RemoveContainer" containerID="9194faf246cb5dfc0fde515b827cbd853ae8db73c167442cb973dc3656e5cdac"
Sep 09 20:35:03.669737 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.669721    2556 scope.go:117] "RemoveContainer" containerID="c751b7ce5b900822f61e4e7c7b1a16fe198f60ad58b378db61034e0f6e3c1867"
Sep 09 20:35:03.677812 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.677794    2556 scope.go:117] "RemoveContainer" containerID="85484e2e9e951ca6cb073b29cf3b3dff41f4bbb22997db2bc3f2ff96925162e4"
Sep 09 20:35:03.678475 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:35:03.678449    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85484e2e9e951ca6cb073b29cf3b3dff41f4bbb22997db2bc3f2ff96925162e4\": container with ID starting with 85484e2e9e951ca6cb073b29cf3b3dff41f4bbb22997db2bc3f2ff96925162e4 not found: ID does not exist" containerID="85484e2e9e951ca6cb073b29cf3b3dff41f4bbb22997db2bc3f2ff96925162e4"
Sep 09 20:35:03.678579 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.678482    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85484e2e9e951ca6cb073b29cf3b3dff41f4bbb22997db2bc3f2ff96925162e4"} err="failed to get container status \"85484e2e9e951ca6cb073b29cf3b3dff41f4bbb22997db2bc3f2ff96925162e4\": rpc error: code = NotFound desc = could not find container \"85484e2e9e951ca6cb073b29cf3b3dff41f4bbb22997db2bc3f2ff96925162e4\": container with ID starting with 85484e2e9e951ca6cb073b29cf3b3dff41f4bbb22997db2bc3f2ff96925162e4 not found: ID does not exist"
Sep 09 20:35:03.678579 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.678515    2556 scope.go:117] "RemoveContainer" containerID="f0543141fc2af09cacddc16db26fda4e4751c57506860b4fce1f9f33b791d637"
Sep 09 20:35:03.678851 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:35:03.678829    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0543141fc2af09cacddc16db26fda4e4751c57506860b4fce1f9f33b791d637\": container with ID starting with f0543141fc2af09cacddc16db26fda4e4751c57506860b4fce1f9f33b791d637 not found: ID does not exist" containerID="f0543141fc2af09cacddc16db26fda4e4751c57506860b4fce1f9f33b791d637"
Sep 09 20:35:03.678952 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.678857    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0543141fc2af09cacddc16db26fda4e4751c57506860b4fce1f9f33b791d637"} err="failed to get container status \"f0543141fc2af09cacddc16db26fda4e4751c57506860b4fce1f9f33b791d637\": rpc error: code = NotFound desc = could not find container \"f0543141fc2af09cacddc16db26fda4e4751c57506860b4fce1f9f33b791d637\": container with ID starting with f0543141fc2af09cacddc16db26fda4e4751c57506860b4fce1f9f33b791d637 not found: ID does not exist"
Sep 09 20:35:03.678952 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.678873    2556 scope.go:117] "RemoveContainer" containerID="7770ed6b37ea9bb43dcafbde34818dd77d62993f0dbf9abe3c2e6b163af2879d"
Sep 09 20:35:03.679176 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:35:03.679157    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7770ed6b37ea9bb43dcafbde34818dd77d62993f0dbf9abe3c2e6b163af2879d\": container with ID starting with 7770ed6b37ea9bb43dcafbde34818dd77d62993f0dbf9abe3c2e6b163af2879d not found: ID does not exist" containerID="7770ed6b37ea9bb43dcafbde34818dd77d62993f0dbf9abe3c2e6b163af2879d"
Sep 09 20:35:03.679219 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.679182    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7770ed6b37ea9bb43dcafbde34818dd77d62993f0dbf9abe3c2e6b163af2879d"} err="failed to get container status \"7770ed6b37ea9bb43dcafbde34818dd77d62993f0dbf9abe3c2e6b163af2879d\": rpc error: code = NotFound desc = could not find container \"7770ed6b37ea9bb43dcafbde34818dd77d62993f0dbf9abe3c2e6b163af2879d\": container with ID starting with 7770ed6b37ea9bb43dcafbde34818dd77d62993f0dbf9abe3c2e6b163af2879d not found: ID does not exist"
Sep 09 20:35:03.679219 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.679206    2556 scope.go:117] "RemoveContainer" containerID="791dd8f32c09dbc4776f552d9ff20d420675a21e27cb2b2162c8118ce869685a"
Sep 09 20:35:03.679488 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:35:03.679470    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"791dd8f32c09dbc4776f552d9ff20d420675a21e27cb2b2162c8118ce869685a\": container with ID starting with 791dd8f32c09dbc4776f552d9ff20d420675a21e27cb2b2162c8118ce869685a not found: ID does not exist" containerID="791dd8f32c09dbc4776f552d9ff20d420675a21e27cb2b2162c8118ce869685a"
Sep 09 20:35:03.679546 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.679493    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"791dd8f32c09dbc4776f552d9ff20d420675a21e27cb2b2162c8118ce869685a"} err="failed to get container status \"791dd8f32c09dbc4776f552d9ff20d420675a21e27cb2b2162c8118ce869685a\": rpc error: code = NotFound desc = could not find container \"791dd8f32c09dbc4776f552d9ff20d420675a21e27cb2b2162c8118ce869685a\": container with ID starting with 791dd8f32c09dbc4776f552d9ff20d420675a21e27cb2b2162c8118ce869685a not found: ID does not exist"
Sep 09 20:35:03.679546 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.679508    2556 scope.go:117] "RemoveContainer" containerID="a0ba094c334cdd4f5f061a05fa13c216d986c5f2fcb8c4cb22a7b3f0ef8ad2c2"
Sep 09 20:35:03.679835 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:35:03.679819    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0ba094c334cdd4f5f061a05fa13c216d986c5f2fcb8c4cb22a7b3f0ef8ad2c2\": container with ID starting with a0ba094c334cdd4f5f061a05fa13c216d986c5f2fcb8c4cb22a7b3f0ef8ad2c2 not found: ID does not exist" containerID="a0ba094c334cdd4f5f061a05fa13c216d986c5f2fcb8c4cb22a7b3f0ef8ad2c2"
Sep 09 20:35:03.679874 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.679838    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ba094c334cdd4f5f061a05fa13c216d986c5f2fcb8c4cb22a7b3f0ef8ad2c2"} err="failed to get container status \"a0ba094c334cdd4f5f061a05fa13c216d986c5f2fcb8c4cb22a7b3f0ef8ad2c2\": rpc error: code = NotFound desc = could not find container \"a0ba094c334cdd4f5f061a05fa13c216d986c5f2fcb8c4cb22a7b3f0ef8ad2c2\": container with ID starting with a0ba094c334cdd4f5f061a05fa13c216d986c5f2fcb8c4cb22a7b3f0ef8ad2c2 not found: ID does not exist"
Sep 09 20:35:03.679874 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.679851    2556 scope.go:117] "RemoveContainer" containerID="9194faf246cb5dfc0fde515b827cbd853ae8db73c167442cb973dc3656e5cdac"
Sep 09 20:35:03.680094 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:35:03.680079    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9194faf246cb5dfc0fde515b827cbd853ae8db73c167442cb973dc3656e5cdac\": container with ID starting with 9194faf246cb5dfc0fde515b827cbd853ae8db73c167442cb973dc3656e5cdac not found: ID does not exist" containerID="9194faf246cb5dfc0fde515b827cbd853ae8db73c167442cb973dc3656e5cdac"
Sep 09 20:35:03.680133 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.680096    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9194faf246cb5dfc0fde515b827cbd853ae8db73c167442cb973dc3656e5cdac"} err="failed to get container status \"9194faf246cb5dfc0fde515b827cbd853ae8db73c167442cb973dc3656e5cdac\": rpc error: code = NotFound desc = could not find container \"9194faf246cb5dfc0fde515b827cbd853ae8db73c167442cb973dc3656e5cdac\": container with ID starting with 9194faf246cb5dfc0fde515b827cbd853ae8db73c167442cb973dc3656e5cdac not found: ID does not exist"
Sep 09 20:35:03.680133 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.680108    2556 scope.go:117] "RemoveContainer" containerID="c751b7ce5b900822f61e4e7c7b1a16fe198f60ad58b378db61034e0f6e3c1867"
Sep 09 20:35:03.680325 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:35:03.680310    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c751b7ce5b900822f61e4e7c7b1a16fe198f60ad58b378db61034e0f6e3c1867\": container with ID starting with c751b7ce5b900822f61e4e7c7b1a16fe198f60ad58b378db61034e0f6e3c1867 not found: ID does not exist" containerID="c751b7ce5b900822f61e4e7c7b1a16fe198f60ad58b378db61034e0f6e3c1867"
Sep 09 20:35:03.680371 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.680328    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c751b7ce5b900822f61e4e7c7b1a16fe198f60ad58b378db61034e0f6e3c1867"} err="failed to get container status \"c751b7ce5b900822f61e4e7c7b1a16fe198f60ad58b378db61034e0f6e3c1867\": rpc error: code = NotFound desc = could not find container \"c751b7ce5b900822f61e4e7c7b1a16fe198f60ad58b378db61034e0f6e3c1867\": container with ID starting with c751b7ce5b900822f61e4e7c7b1a16fe198f60ad58b378db61034e0f6e3c1867 not found: ID does not exist"
Sep 09 20:35:03.680371 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.680339    2556 scope.go:117] "RemoveContainer" containerID="85484e2e9e951ca6cb073b29cf3b3dff41f4bbb22997db2bc3f2ff96925162e4"
Sep 09 20:35:03.680556 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.680535    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85484e2e9e951ca6cb073b29cf3b3dff41f4bbb22997db2bc3f2ff96925162e4"} err="failed to get container status \"85484e2e9e951ca6cb073b29cf3b3dff41f4bbb22997db2bc3f2ff96925162e4\": rpc error: code = NotFound desc = could not find container \"85484e2e9e951ca6cb073b29cf3b3dff41f4bbb22997db2bc3f2ff96925162e4\": container with ID starting with 85484e2e9e951ca6cb073b29cf3b3dff41f4bbb22997db2bc3f2ff96925162e4 not found: ID does not exist"
Sep 09 20:35:03.680626 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.680558    2556 scope.go:117] "RemoveContainer" containerID="f0543141fc2af09cacddc16db26fda4e4751c57506860b4fce1f9f33b791d637"
Sep 09 20:35:03.680780 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.680764    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0543141fc2af09cacddc16db26fda4e4751c57506860b4fce1f9f33b791d637"} err="failed to get container status \"f0543141fc2af09cacddc16db26fda4e4751c57506860b4fce1f9f33b791d637\": rpc error: code = NotFound desc = could not find container \"f0543141fc2af09cacddc16db26fda4e4751c57506860b4fce1f9f33b791d637\": container with ID starting with f0543141fc2af09cacddc16db26fda4e4751c57506860b4fce1f9f33b791d637 not found: ID does not exist"
Sep 09 20:35:03.680825 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.680781    2556 scope.go:117] "RemoveContainer" containerID="7770ed6b37ea9bb43dcafbde34818dd77d62993f0dbf9abe3c2e6b163af2879d"
Sep 09 20:35:03.681017 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.680997    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7770ed6b37ea9bb43dcafbde34818dd77d62993f0dbf9abe3c2e6b163af2879d"} err="failed to get container status \"7770ed6b37ea9bb43dcafbde34818dd77d62993f0dbf9abe3c2e6b163af2879d\": rpc error: code = NotFound desc = could not find container \"7770ed6b37ea9bb43dcafbde34818dd77d62993f0dbf9abe3c2e6b163af2879d\": container with ID starting with 7770ed6b37ea9bb43dcafbde34818dd77d62993f0dbf9abe3c2e6b163af2879d not found: ID does not exist"
Sep 09 20:35:03.681089 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.681018    2556 scope.go:117] "RemoveContainer" containerID="791dd8f32c09dbc4776f552d9ff20d420675a21e27cb2b2162c8118ce869685a"
Sep 09 20:35:03.681261 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.681245    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"791dd8f32c09dbc4776f552d9ff20d420675a21e27cb2b2162c8118ce869685a"} err="failed to get container status \"791dd8f32c09dbc4776f552d9ff20d420675a21e27cb2b2162c8118ce869685a\": rpc error: code = NotFound desc = could not find container \"791dd8f32c09dbc4776f552d9ff20d420675a21e27cb2b2162c8118ce869685a\": container with ID starting with 791dd8f32c09dbc4776f552d9ff20d420675a21e27cb2b2162c8118ce869685a not found: ID does not exist"
Sep 09 20:35:03.681308 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.681262    2556 scope.go:117] "RemoveContainer" containerID="a0ba094c334cdd4f5f061a05fa13c216d986c5f2fcb8c4cb22a7b3f0ef8ad2c2"
Sep 09 20:35:03.681461 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.681443    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ba094c334cdd4f5f061a05fa13c216d986c5f2fcb8c4cb22a7b3f0ef8ad2c2"} err="failed to get container status \"a0ba094c334cdd4f5f061a05fa13c216d986c5f2fcb8c4cb22a7b3f0ef8ad2c2\": rpc error: code = NotFound desc = could not find container \"a0ba094c334cdd4f5f061a05fa13c216d986c5f2fcb8c4cb22a7b3f0ef8ad2c2\": container with ID starting with a0ba094c334cdd4f5f061a05fa13c216d986c5f2fcb8c4cb22a7b3f0ef8ad2c2 not found: ID does not exist"
Sep 09 20:35:03.681505 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.681461    2556 scope.go:117] "RemoveContainer" containerID="9194faf246cb5dfc0fde515b827cbd853ae8db73c167442cb973dc3656e5cdac"
Sep 09 20:35:03.681640 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.681625    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9194faf246cb5dfc0fde515b827cbd853ae8db73c167442cb973dc3656e5cdac"} err="failed to get container status \"9194faf246cb5dfc0fde515b827cbd853ae8db73c167442cb973dc3656e5cdac\": rpc error: code = NotFound desc = could not find container \"9194faf246cb5dfc0fde515b827cbd853ae8db73c167442cb973dc3656e5cdac\": container with ID starting with 9194faf246cb5dfc0fde515b827cbd853ae8db73c167442cb973dc3656e5cdac not found: ID does not exist"
Sep 09 20:35:03.681680 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.681639    2556 scope.go:117] "RemoveContainer" containerID="c751b7ce5b900822f61e4e7c7b1a16fe198f60ad58b378db61034e0f6e3c1867"
Sep 09 20:35:03.681820 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.681805    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c751b7ce5b900822f61e4e7c7b1a16fe198f60ad58b378db61034e0f6e3c1867"} err="failed to get container status \"c751b7ce5b900822f61e4e7c7b1a16fe198f60ad58b378db61034e0f6e3c1867\": rpc error: code = NotFound desc = could not find container \"c751b7ce5b900822f61e4e7c7b1a16fe198f60ad58b378db61034e0f6e3c1867\": container with ID starting with c751b7ce5b900822f61e4e7c7b1a16fe198f60ad58b378db61034e0f6e3c1867 not found: ID does not exist"
Sep 09 20:35:03.687937 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.687892    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:35:03.688197 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.688184    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="alertmanager"
Sep 09 20:35:03.688233 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.688198    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="alertmanager"
Sep 09 20:35:03.688233 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.688212    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="prom-label-proxy"
Sep 09 20:35:03.688233 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.688218    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="prom-label-proxy"
Sep 09 20:35:03.688233 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.688226    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="init-config-reloader"
Sep 09 20:35:03.688233 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.688232    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="init-config-reloader"
Sep 09 20:35:03.688369 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.688239    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="config-reloader"
Sep 09 20:35:03.688369 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.688244    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="config-reloader"
Sep 09 20:35:03.688369 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.688251    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="kube-rbac-proxy-web"
Sep 09 20:35:03.688369 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.688257    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="kube-rbac-proxy-web"
Sep 09 20:35:03.688369 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.688262    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="kube-rbac-proxy"
Sep 09 20:35:03.688369 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.688268    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="kube-rbac-proxy"
Sep 09 20:35:03.688369 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.688275    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="kube-rbac-proxy-metric"
Sep 09 20:35:03.688369 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.688281    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="kube-rbac-proxy-metric"
Sep 09 20:35:03.688369 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.688326    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="alertmanager"
Sep 09 20:35:03.688369 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.688334    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="kube-rbac-proxy-web"
Sep 09 20:35:03.688369 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.688341    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="prom-label-proxy"
Sep 09 20:35:03.688369 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.688349    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="config-reloader"
Sep 09 20:35:03.688369 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.688355    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="kube-rbac-proxy-metric"
Sep 09 20:35:03.688369 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.688360    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" containerName="kube-rbac-proxy"
Sep 09 20:35:03.691745 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.691730    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.696283 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.696260    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 20:35:03.696378 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.696286    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 20:35:03.696378 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.696365    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 20:35:03.696494 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.696371    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 20:35:03.696494 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.696365    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 20:35:03.696616 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.696600    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 20:35:03.696702 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.696672    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 20:35:03.696759 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.696729    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 20:35:03.696816 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.696758    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8j7w6\""
Sep 09 20:35:03.702400 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.702379    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 20:35:03.706986 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.706968    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:35:03.786863 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.786801    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7607422d-c966-40af-a7ff-3e3e854743c2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.786863 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.786829    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7607422d-c966-40af-a7ff-3e3e854743c2-config-volume\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.786863 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.786847    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7607422d-c966-40af-a7ff-3e3e854743c2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.787087 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.786864    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7607422d-c966-40af-a7ff-3e3e854743c2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.787087 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.786965    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7607422d-c966-40af-a7ff-3e3e854743c2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.787087 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.786999    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7607422d-c966-40af-a7ff-3e3e854743c2-config-out\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.787087 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.787015    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7607422d-c966-40af-a7ff-3e3e854743c2-web-config\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.787087 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.787044    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-945rj\" (UniqueName: \"kubernetes.io/projected/7607422d-c966-40af-a7ff-3e3e854743c2-kube-api-access-945rj\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.787329 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.787089    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7607422d-c966-40af-a7ff-3e3e854743c2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.787329 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.787122    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7607422d-c966-40af-a7ff-3e3e854743c2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.787329 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.787159    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7607422d-c966-40af-a7ff-3e3e854743c2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.787329 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.787206    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7607422d-c966-40af-a7ff-3e3e854743c2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.787329 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.787252    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7607422d-c966-40af-a7ff-3e3e854743c2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.887771 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.887747    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7607422d-c966-40af-a7ff-3e3e854743c2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.887771 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.887772    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7607422d-c966-40af-a7ff-3e3e854743c2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.887892 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.887794    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7607422d-c966-40af-a7ff-3e3e854743c2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.887892 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.887825    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7607422d-c966-40af-a7ff-3e3e854743c2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.887892 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.887859    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7607422d-c966-40af-a7ff-3e3e854743c2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.887892 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.887887    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7607422d-c966-40af-a7ff-3e3e854743c2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.888101 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.887935    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7607422d-c966-40af-a7ff-3e3e854743c2-config-volume\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.888101 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.887957    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7607422d-c966-40af-a7ff-3e3e854743c2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.888101 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.887984    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7607422d-c966-40af-a7ff-3e3e854743c2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.888101 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.888028    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7607422d-c966-40af-a7ff-3e3e854743c2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.888101 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.888065    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7607422d-c966-40af-a7ff-3e3e854743c2-config-out\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.888101 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.888090    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7607422d-c966-40af-a7ff-3e3e854743c2-web-config\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.888400 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.888130    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-945rj\" (UniqueName: \"kubernetes.io/projected/7607422d-c966-40af-a7ff-3e3e854743c2-kube-api-access-945rj\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.888400 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.888225    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7607422d-c966-40af-a7ff-3e3e854743c2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.888972 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.888894    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7607422d-c966-40af-a7ff-3e3e854743c2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.890954 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.890926    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7607422d-c966-40af-a7ff-3e3e854743c2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.891060 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.890953    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7607422d-c966-40af-a7ff-3e3e854743c2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.891060 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.890973    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7607422d-c966-40af-a7ff-3e3e854743c2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.891180 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.891062    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7607422d-c966-40af-a7ff-3e3e854743c2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.891232 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.891196    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7607422d-c966-40af-a7ff-3e3e854743c2-config-volume\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.891232 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.891197    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7607422d-c966-40af-a7ff-3e3e854743c2-config-out\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.891332 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.891271    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7607422d-c966-40af-a7ff-3e3e854743c2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.891414 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.891395    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7607422d-c966-40af-a7ff-3e3e854743c2-web-config\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.891676 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.891653    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7607422d-c966-40af-a7ff-3e3e854743c2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.892792 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.892776    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7607422d-c966-40af-a7ff-3e3e854743c2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:03.897732 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:03.897715    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-945rj\" (UniqueName: \"kubernetes.io/projected/7607422d-c966-40af-a7ff-3e3e854743c2-kube-api-access-945rj\") pod \"alertmanager-main-0\" (UID: \"7607422d-c966-40af-a7ff-3e3e854743c2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:04.001997 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:04.001972    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:35:04.281244 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:04.281218    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:35:04.282698 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:35:04.282676    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7607422d_c966_40af_a7ff_3e3e854743c2.slice/crio-d1a2a61e9caab751937198371166e5ba69c5fb22d64096fab689b7f81df63f37 WatchSource:0}: Error finding container d1a2a61e9caab751937198371166e5ba69c5fb22d64096fab689b7f81df63f37: Status 404 returned error can't find the container with id d1a2a61e9caab751937198371166e5ba69c5fb22d64096fab689b7f81df63f37
Sep 09 20:35:04.631758 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:04.631682    2556 generic.go:356] "Generic (PLEG): container finished" podID="7607422d-c966-40af-a7ff-3e3e854743c2" containerID="6bc992baeca407444e5c54ac8a0fe3b21b7c26abbc481005900b6f87556f91c9" exitCode=0
Sep 09 20:35:04.632165 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:04.631772    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7607422d-c966-40af-a7ff-3e3e854743c2","Type":"ContainerDied","Data":"6bc992baeca407444e5c54ac8a0fe3b21b7c26abbc481005900b6f87556f91c9"}
Sep 09 20:35:04.632165 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:04.631853    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7607422d-c966-40af-a7ff-3e3e854743c2","Type":"ContainerStarted","Data":"d1a2a61e9caab751937198371166e5ba69c5fb22d64096fab689b7f81df63f37"}
Sep 09 20:35:04.892825 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:04.892803    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e972a506-9e4a-44dd-8924-1e56ad707ac6" path="/var/lib/kubelet/pods/e972a506-9e4a-44dd-8924-1e56ad707ac6/volumes"
Sep 09 20:35:05.640136 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:05.640105    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7607422d-c966-40af-a7ff-3e3e854743c2","Type":"ContainerStarted","Data":"f076c264822fca6c73565f16a8fbf77b4afcb8e3a5d9783f1a9a9af0388f17e9"}
Sep 09 20:35:05.640481 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:05.640165    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7607422d-c966-40af-a7ff-3e3e854743c2","Type":"ContainerStarted","Data":"eba19a60c21d5b908df49185b2fd297cd7ddd9c73ae7b49c5324a2df58ae93c1"}
Sep 09 20:35:05.640481 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:05.640191    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7607422d-c966-40af-a7ff-3e3e854743c2","Type":"ContainerStarted","Data":"7a3e979448d93df7630cfd8bc73c866e93ff161a37cfd574857e9a6a70778a91"}
Sep 09 20:35:05.640481 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:05.640215    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7607422d-c966-40af-a7ff-3e3e854743c2","Type":"ContainerStarted","Data":"d7a23bf02c6be55392079d45200f96fe002b6f6a3e1e0d3e6b0d5772f75e1af4"}
Sep 09 20:35:05.640481 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:05.640238    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7607422d-c966-40af-a7ff-3e3e854743c2","Type":"ContainerStarted","Data":"f0cc2f54ca59f7b70c513644a9b47a98bc3df0b5b01f19bc1f52e02a2a990b8b"}
Sep 09 20:35:05.640481 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:05.640260    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7607422d-c966-40af-a7ff-3e3e854743c2","Type":"ContainerStarted","Data":"1a6ac7dde2c303d02d95736ee2daba86b3ed4dd6ede6928701322f70b97c34fb"}
Sep 09 20:35:05.673834 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:05.673785    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.655752624 podStartE2EDuration="2.673769655s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.017031ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:35:05.671433072 +0000 UTC m=+151.351393384" watchObservedRunningTime="2026-09-09 20:35:05.673769655 +0000 UTC m=+151.353729964"
Sep 09 20:35:06.427674 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.427639    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:35:06.428059 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.428026    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="prometheus" containerID="cri-o://992ad2e864b99ebef689e51d5732730c32c1a091c44826cadec41bde4878b0eb" gracePeriod=600
Sep 09 20:35:06.428166 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.428066    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b4cbaff0dcf0d205880edb2a2301cd3a399b5767dbc952d6a3713058ce82364d" gracePeriod=600
Sep 09 20:35:06.428166 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.428084    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="config-reloader" containerID="cri-o://a7259e847e92e1956caa72606b12bcbb1fec2162c9793d2ccc9520d153a44b20" gracePeriod=600
Sep 09 20:35:06.428166 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.428085    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="kube-rbac-proxy-web" containerID="cri-o://a8b25005a7a0eea4db6f01967dd87ec29d904aeb405b16663d2ad8e692c93f63" gracePeriod=600
Sep 09 20:35:06.428322 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.428070    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="thanos-sidecar" containerID="cri-o://528a4219cf514afb314003c0e2d89d0992be6d850752c7f95df502bce963de03" gracePeriod=600
Sep 09 20:35:06.428322 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.428036    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="kube-rbac-proxy" containerID="cri-o://fbd0aac93c2e44484ed9092c7c421b15f229218bae4c1b45045656736054ab06" gracePeriod=600
Sep 09 20:35:06.650867 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.650835    2556 generic.go:356] "Generic (PLEG): container finished" podID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerID="b4cbaff0dcf0d205880edb2a2301cd3a399b5767dbc952d6a3713058ce82364d" exitCode=0
Sep 09 20:35:06.650867 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.650861    2556 generic.go:356] "Generic (PLEG): container finished" podID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerID="fbd0aac93c2e44484ed9092c7c421b15f229218bae4c1b45045656736054ab06" exitCode=0
Sep 09 20:35:06.650867 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.650867    2556 generic.go:356] "Generic (PLEG): container finished" podID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerID="a8b25005a7a0eea4db6f01967dd87ec29d904aeb405b16663d2ad8e692c93f63" exitCode=0
Sep 09 20:35:06.650867 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.650874    2556 generic.go:356] "Generic (PLEG): container finished" podID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerID="528a4219cf514afb314003c0e2d89d0992be6d850752c7f95df502bce963de03" exitCode=0
Sep 09 20:35:06.651341 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.650879    2556 generic.go:356] "Generic (PLEG): container finished" podID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerID="a7259e847e92e1956caa72606b12bcbb1fec2162c9793d2ccc9520d153a44b20" exitCode=0
Sep 09 20:35:06.651341 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.650884    2556 generic.go:356] "Generic (PLEG): container finished" podID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerID="992ad2e864b99ebef689e51d5732730c32c1a091c44826cadec41bde4878b0eb" exitCode=0
Sep 09 20:35:06.651341 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.650945    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06acd67c-1f5b-4053-9aa9-5147c56411e4","Type":"ContainerDied","Data":"b4cbaff0dcf0d205880edb2a2301cd3a399b5767dbc952d6a3713058ce82364d"}
Sep 09 20:35:06.651341 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.651036    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06acd67c-1f5b-4053-9aa9-5147c56411e4","Type":"ContainerDied","Data":"fbd0aac93c2e44484ed9092c7c421b15f229218bae4c1b45045656736054ab06"}
Sep 09 20:35:06.651341 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.651084    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06acd67c-1f5b-4053-9aa9-5147c56411e4","Type":"ContainerDied","Data":"a8b25005a7a0eea4db6f01967dd87ec29d904aeb405b16663d2ad8e692c93f63"}
Sep 09 20:35:06.651341 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.651126    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06acd67c-1f5b-4053-9aa9-5147c56411e4","Type":"ContainerDied","Data":"528a4219cf514afb314003c0e2d89d0992be6d850752c7f95df502bce963de03"}
Sep 09 20:35:06.651341 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.651166    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06acd67c-1f5b-4053-9aa9-5147c56411e4","Type":"ContainerDied","Data":"a7259e847e92e1956caa72606b12bcbb1fec2162c9793d2ccc9520d153a44b20"}
Sep 09 20:35:06.651341 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.651206    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06acd67c-1f5b-4053-9aa9-5147c56411e4","Type":"ContainerDied","Data":"992ad2e864b99ebef689e51d5732730c32c1a091c44826cadec41bde4878b0eb"}
Sep 09 20:35:06.672422 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.672400    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:06.815356 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.815314    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-trusted-ca-bundle\") pod \"06acd67c-1f5b-4053-9aa9-5147c56411e4\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") "
Sep 09 20:35:06.815356 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.815362    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-rulefiles-1\") pod \"06acd67c-1f5b-4053-9aa9-5147c56411e4\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") "
Sep 09 20:35:06.815568 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.815397    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-db\") pod \"06acd67c-1f5b-4053-9aa9-5147c56411e4\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") "
Sep 09 20:35:06.815568 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.815434    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-web-config\") pod \"06acd67c-1f5b-4053-9aa9-5147c56411e4\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") "
Sep 09 20:35:06.815568 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.815474    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06acd67c-1f5b-4053-9aa9-5147c56411e4-config-out\") pod \"06acd67c-1f5b-4053-9aa9-5147c56411e4\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") "
Sep 09 20:35:06.815568 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.815509    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"06acd67c-1f5b-4053-9aa9-5147c56411e4\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") "
Sep 09 20:35:06.815568 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.815552    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-configmap-kubelet-serving-ca-bundle\") pod \"06acd67c-1f5b-4053-9aa9-5147c56411e4\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") "
Sep 09 20:35:06.815806 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.815576    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-rulefiles-2\") pod \"06acd67c-1f5b-4053-9aa9-5147c56411e4\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") "
Sep 09 20:35:06.815806 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.815603    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jh42r\" (UniqueName: \"kubernetes.io/projected/06acd67c-1f5b-4053-9aa9-5147c56411e4-kube-api-access-jh42r\") pod \"06acd67c-1f5b-4053-9aa9-5147c56411e4\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") "
Sep 09 20:35:06.815806 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.815633    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-config\") pod \"06acd67c-1f5b-4053-9aa9-5147c56411e4\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") "
Sep 09 20:35:06.815806 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.815666    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-thanos-prometheus-http-client-file\") pod \"06acd67c-1f5b-4053-9aa9-5147c56411e4\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") "
Sep 09 20:35:06.815806 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.815691    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-kube-rbac-proxy\") pod \"06acd67c-1f5b-4053-9aa9-5147c56411e4\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") "
Sep 09 20:35:06.815806 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.815694    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "06acd67c-1f5b-4053-9aa9-5147c56411e4" (UID: "06acd67c-1f5b-4053-9aa9-5147c56411e4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:06.815806 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.815743    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-configmap-serving-certs-ca-bundle\") pod \"06acd67c-1f5b-4053-9aa9-5147c56411e4\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") "
Sep 09 20:35:06.815806 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.815746    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "06acd67c-1f5b-4053-9aa9-5147c56411e4" (UID: "06acd67c-1f5b-4053-9aa9-5147c56411e4"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:06.815806 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.815772    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06acd67c-1f5b-4053-9aa9-5147c56411e4-tls-assets\") pod \"06acd67c-1f5b-4053-9aa9-5147c56411e4\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") "
Sep 09 20:35:06.815806 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.815807    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"06acd67c-1f5b-4053-9aa9-5147c56411e4\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") "
Sep 09 20:35:06.816385 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.815853    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-metrics-client-certs\") pod \"06acd67c-1f5b-4053-9aa9-5147c56411e4\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") "
Sep 09 20:35:06.816385 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.815878    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-grpc-tls\") pod \"06acd67c-1f5b-4053-9aa9-5147c56411e4\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") "
Sep 09 20:35:06.816385 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.815934    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-rulefiles-0\") pod \"06acd67c-1f5b-4053-9aa9-5147c56411e4\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") "
Sep 09 20:35:06.816385 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.815938    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "06acd67c-1f5b-4053-9aa9-5147c56411e4" (UID: "06acd67c-1f5b-4053-9aa9-5147c56411e4"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:06.816385 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.815963    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-configmap-metrics-client-ca\") pod \"06acd67c-1f5b-4053-9aa9-5147c56411e4\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") "
Sep 09 20:35:06.816385 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.815990    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-prometheus-k8s-tls\") pod \"06acd67c-1f5b-4053-9aa9-5147c56411e4\" (UID: \"06acd67c-1f5b-4053-9aa9-5147c56411e4\") "
Sep 09 20:35:06.816385 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.816283    2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:06.816385 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.816303    2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:06.816385 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.816317    2556 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:06.816845 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.816405    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "06acd67c-1f5b-4053-9aa9-5147c56411e4" (UID: "06acd67c-1f5b-4053-9aa9-5147c56411e4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:06.816845 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.816449    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "06acd67c-1f5b-4053-9aa9-5147c56411e4" (UID: "06acd67c-1f5b-4053-9aa9-5147c56411e4"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:06.818221 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.817994    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06acd67c-1f5b-4053-9aa9-5147c56411e4-config-out" (OuterVolumeSpecName: "config-out") pod "06acd67c-1f5b-4053-9aa9-5147c56411e4" (UID: "06acd67c-1f5b-4053-9aa9-5147c56411e4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:06.818221 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.818045    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "06acd67c-1f5b-4053-9aa9-5147c56411e4" (UID: "06acd67c-1f5b-4053-9aa9-5147c56411e4"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:06.818409 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.818242    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-config" (OuterVolumeSpecName: "config") pod "06acd67c-1f5b-4053-9aa9-5147c56411e4" (UID: "06acd67c-1f5b-4053-9aa9-5147c56411e4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:06.818609 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.818582    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "06acd67c-1f5b-4053-9aa9-5147c56411e4" (UID: "06acd67c-1f5b-4053-9aa9-5147c56411e4"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:06.818609 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.818585    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "06acd67c-1f5b-4053-9aa9-5147c56411e4" (UID: "06acd67c-1f5b-4053-9aa9-5147c56411e4"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:06.818960 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.818937    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06acd67c-1f5b-4053-9aa9-5147c56411e4-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "06acd67c-1f5b-4053-9aa9-5147c56411e4" (UID: "06acd67c-1f5b-4053-9aa9-5147c56411e4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:06.819217 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.819181    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "06acd67c-1f5b-4053-9aa9-5147c56411e4" (UID: "06acd67c-1f5b-4053-9aa9-5147c56411e4"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:06.819555 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.819528    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "06acd67c-1f5b-4053-9aa9-5147c56411e4" (UID: "06acd67c-1f5b-4053-9aa9-5147c56411e4"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:06.819814 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.819787    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06acd67c-1f5b-4053-9aa9-5147c56411e4-kube-api-access-jh42r" (OuterVolumeSpecName: "kube-api-access-jh42r") pod "06acd67c-1f5b-4053-9aa9-5147c56411e4" (UID: "06acd67c-1f5b-4053-9aa9-5147c56411e4"). InnerVolumeSpecName "kube-api-access-jh42r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:06.820025 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.820001    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "06acd67c-1f5b-4053-9aa9-5147c56411e4" (UID: "06acd67c-1f5b-4053-9aa9-5147c56411e4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:06.820247 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.820224    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "06acd67c-1f5b-4053-9aa9-5147c56411e4" (UID: "06acd67c-1f5b-4053-9aa9-5147c56411e4"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:06.820308 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.820239    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "06acd67c-1f5b-4053-9aa9-5147c56411e4" (UID: "06acd67c-1f5b-4053-9aa9-5147c56411e4"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:06.820580 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.820561    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "06acd67c-1f5b-4053-9aa9-5147c56411e4" (UID: "06acd67c-1f5b-4053-9aa9-5147c56411e4"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:06.820818 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.820801    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "06acd67c-1f5b-4053-9aa9-5147c56411e4" (UID: "06acd67c-1f5b-4053-9aa9-5147c56411e4"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:06.829035 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.829015    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-web-config" (OuterVolumeSpecName: "web-config") pod "06acd67c-1f5b-4053-9aa9-5147c56411e4" (UID: "06acd67c-1f5b-4053-9aa9-5147c56411e4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:06.916647 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.916625    2556 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-metrics-client-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:06.916647 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.916646    2556 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-grpc-tls\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:06.916763 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.916655    2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:06.916763 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.916663    2556 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-configmap-metrics-client-ca\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:06.916763 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.916672    2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:06.916763 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.916682    2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-db\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:06.916763 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.916691    2556 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-web-config\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:06.916763 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.916700    2556 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06acd67c-1f5b-4053-9aa9-5147c56411e4-config-out\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:06.916763 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.916708    2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:06.916763 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.916717    2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:06.916763 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.916725    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh42r\" (UniqueName: \"kubernetes.io/projected/06acd67c-1f5b-4053-9aa9-5147c56411e4-kube-api-access-jh42r\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:06.916763 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.916733    2556 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-config\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:06.916763 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.916743    2556 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:06.916763 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.916751    2556 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-kube-rbac-proxy\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:06.916763 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.916760    2556 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06acd67c-1f5b-4053-9aa9-5147c56411e4-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:06.917180 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.916770    2556 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06acd67c-1f5b-4053-9aa9-5147c56411e4-tls-assets\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:06.917180 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:06.916778    2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06acd67c-1f5b-4053-9aa9-5147c56411e4-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:35:07.658823 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.658792    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06acd67c-1f5b-4053-9aa9-5147c56411e4","Type":"ContainerDied","Data":"3d8bcc85e0ada2cf47fc182d05ba4a17ba1a72dcec379c90daafbe570ee7df9a"}
Sep 09 20:35:07.659278 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.658860    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.659278 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.658868    2556 scope.go:117] "RemoveContainer" containerID="b4cbaff0dcf0d205880edb2a2301cd3a399b5767dbc952d6a3713058ce82364d"
Sep 09 20:35:07.666708 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.666691    2556 scope.go:117] "RemoveContainer" containerID="fbd0aac93c2e44484ed9092c7c421b15f229218bae4c1b45045656736054ab06"
Sep 09 20:35:07.673521 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.673505    2556 scope.go:117] "RemoveContainer" containerID="a8b25005a7a0eea4db6f01967dd87ec29d904aeb405b16663d2ad8e692c93f63"
Sep 09 20:35:07.681965 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.681944    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:35:07.683626 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.683609    2556 scope.go:117] "RemoveContainer" containerID="528a4219cf514afb314003c0e2d89d0992be6d850752c7f95df502bce963de03"
Sep 09 20:35:07.687955 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.687932    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:35:07.691208 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.691190    2556 scope.go:117] "RemoveContainer" containerID="a7259e847e92e1956caa72606b12bcbb1fec2162c9793d2ccc9520d153a44b20"
Sep 09 20:35:07.697354 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.697338    2556 scope.go:117] "RemoveContainer" containerID="992ad2e864b99ebef689e51d5732730c32c1a091c44826cadec41bde4878b0eb"
Sep 09 20:35:07.704125 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.704108    2556 scope.go:117] "RemoveContainer" containerID="634e954d3f23d811e12adb999919356ae5cc43d619a2d8d3948e3764f3a83ae9"
Sep 09 20:35:07.713288 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.713267    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:35:07.713596 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.713580    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="kube-rbac-proxy-web"
Sep 09 20:35:07.713666 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.713599    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="kube-rbac-proxy-web"
Sep 09 20:35:07.713666 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.713616    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="init-config-reloader"
Sep 09 20:35:07.713666 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.713624    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="init-config-reloader"
Sep 09 20:35:07.713666 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.713639    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="kube-rbac-proxy-thanos"
Sep 09 20:35:07.713666 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.713648    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="kube-rbac-proxy-thanos"
Sep 09 20:35:07.713666 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.713660    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="config-reloader"
Sep 09 20:35:07.714015 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.713668    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="config-reloader"
Sep 09 20:35:07.714015 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.713682    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="kube-rbac-proxy"
Sep 09 20:35:07.714015 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.713691    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="kube-rbac-proxy"
Sep 09 20:35:07.714015 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.713703    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="prometheus"
Sep 09 20:35:07.714015 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.713711    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="prometheus"
Sep 09 20:35:07.714015 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.713726    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="thanos-sidecar"
Sep 09 20:35:07.714015 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.713734    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="thanos-sidecar"
Sep 09 20:35:07.714015 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.713826    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="kube-rbac-proxy-web"
Sep 09 20:35:07.714015 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.713837    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="kube-rbac-proxy-thanos"
Sep 09 20:35:07.714015 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.713848    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="prometheus"
Sep 09 20:35:07.714015 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.713859    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="thanos-sidecar"
Sep 09 20:35:07.714015 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.713868    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="kube-rbac-proxy"
Sep 09 20:35:07.714015 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.713877    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" containerName="config-reloader"
Sep 09 20:35:07.719182 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.718788    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.722002 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.721983    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lhhrg\""
Sep 09 20:35:07.722634 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.722614    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 20:35:07.722634 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.722628    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 20:35:07.722860 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.722630    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 20:35:07.722860 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.722618    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 20:35:07.722860 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.722734    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 20:35:07.723078 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.722970    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 20:35:07.723078 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.723023    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 20:35:07.723179 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.723084    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 20:35:07.723179 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.722971    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 20:35:07.723610 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.723553    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 20:35:07.723610 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.723570    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3fe50p7bjlk4k\""
Sep 09 20:35:07.723610 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.723589    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 20:35:07.723985 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.723796    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 20:35:07.725742 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.725722    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 20:35:07.728257 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.728241    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 20:35:07.734244 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.734225    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:35:07.823087 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.823060    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z5qn\" (UniqueName: \"kubernetes.io/projected/82880fb9-9943-4d5d-b23e-b5c3c05319fc-kube-api-access-6z5qn\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.823246 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.823092    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82880fb9-9943-4d5d-b23e-b5c3c05319fc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.823246 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.823110    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.823246 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.823127    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/82880fb9-9943-4d5d-b23e-b5c3c05319fc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.823246 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.823151    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82880fb9-9943-4d5d-b23e-b5c3c05319fc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.823246 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.823197    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82880fb9-9943-4d5d-b23e-b5c3c05319fc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.823246 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.823211    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-config\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.823246 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.823232    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/82880fb9-9943-4d5d-b23e-b5c3c05319fc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.823563 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.823303    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.823563 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.823351    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82880fb9-9943-4d5d-b23e-b5c3c05319fc-config-out\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.823563 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.823394    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82880fb9-9943-4d5d-b23e-b5c3c05319fc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.823563 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.823425    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/82880fb9-9943-4d5d-b23e-b5c3c05319fc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.823563 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.823452    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.823563 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.823478    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.823563 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.823491    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82880fb9-9943-4d5d-b23e-b5c3c05319fc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.823563 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.823506    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/82880fb9-9943-4d5d-b23e-b5c3c05319fc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.823563 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.823537    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-web-config\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.823812 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.823568    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.823812 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.823584    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.823812 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.823603    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.924235 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.924210    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.924336 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.924238    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.924336 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.924259    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.924336 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.924287    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z5qn\" (UniqueName: \"kubernetes.io/projected/82880fb9-9943-4d5d-b23e-b5c3c05319fc-kube-api-access-6z5qn\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.924336 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.924310    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82880fb9-9943-4d5d-b23e-b5c3c05319fc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.924530 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.924334    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.924530 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.924358    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/82880fb9-9943-4d5d-b23e-b5c3c05319fc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.924530 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.924380    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82880fb9-9943-4d5d-b23e-b5c3c05319fc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.924530 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.924409    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82880fb9-9943-4d5d-b23e-b5c3c05319fc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.925278 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.925249    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82880fb9-9943-4d5d-b23e-b5c3c05319fc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.925371 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.925327    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-config\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.925427 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.925368    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/82880fb9-9943-4d5d-b23e-b5c3c05319fc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.925427 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.925406    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.925544 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.925437    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82880fb9-9943-4d5d-b23e-b5c3c05319fc-config-out\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.925544 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.925474    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82880fb9-9943-4d5d-b23e-b5c3c05319fc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.925544 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.925500    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/82880fb9-9943-4d5d-b23e-b5c3c05319fc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.925544 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.925523    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.925743 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.925553    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.925743 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.925579    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82880fb9-9943-4d5d-b23e-b5c3c05319fc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.925743 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.925603    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/82880fb9-9943-4d5d-b23e-b5c3c05319fc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.925743 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.925639    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-web-config\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.926000 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.925812    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82880fb9-9943-4d5d-b23e-b5c3c05319fc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.926747 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.926444    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82880fb9-9943-4d5d-b23e-b5c3c05319fc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.927497 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.927468    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.927497 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.927485    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.927642 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.927501    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82880fb9-9943-4d5d-b23e-b5c3c05319fc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.927642 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.927472    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.928164 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.928145    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-web-config\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.928865 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.928562    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82880fb9-9943-4d5d-b23e-b5c3c05319fc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.928865 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.928639    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/82880fb9-9943-4d5d-b23e-b5c3c05319fc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.928865 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.928762    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/82880fb9-9943-4d5d-b23e-b5c3c05319fc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.928865 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.928832    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/82880fb9-9943-4d5d-b23e-b5c3c05319fc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.928865 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.928864    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.929304 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.929272    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/82880fb9-9943-4d5d-b23e-b5c3c05319fc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.929766 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.929739    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-config\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.930060 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.930038    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.930221 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.930202    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.930711 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.930675    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/82880fb9-9943-4d5d-b23e-b5c3c05319fc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.930711 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.930703    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82880fb9-9943-4d5d-b23e-b5c3c05319fc-config-out\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:07.932504 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:07.932487    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z5qn\" (UniqueName: \"kubernetes.io/projected/82880fb9-9943-4d5d-b23e-b5c3c05319fc-kube-api-access-6z5qn\") pod \"prometheus-k8s-0\" (UID: \"82880fb9-9943-4d5d-b23e-b5c3c05319fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:08.029689 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:08.029663    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:35:08.330618 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:08.330595    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:35:08.332492 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:35:08.332463    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82880fb9_9943_4d5d_b23e_b5c3c05319fc.slice/crio-fec338f1252229aed053558be33d306c7188413c8af2856bfa7a123c4ac9ecd2 WatchSource:0}: Error finding container fec338f1252229aed053558be33d306c7188413c8af2856bfa7a123c4ac9ecd2: Status 404 returned error can't find the container with id fec338f1252229aed053558be33d306c7188413c8af2856bfa7a123c4ac9ecd2
Sep 09 20:35:08.665245 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:08.665178    2556 generic.go:356] "Generic (PLEG): container finished" podID="82880fb9-9943-4d5d-b23e-b5c3c05319fc" containerID="7c92161b8904371775006ed7f46d1299fd504a78cbbfd6e0c224a0fc56da1205" exitCode=0
Sep 09 20:35:08.665245 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:08.665215    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82880fb9-9943-4d5d-b23e-b5c3c05319fc","Type":"ContainerDied","Data":"7c92161b8904371775006ed7f46d1299fd504a78cbbfd6e0c224a0fc56da1205"}
Sep 09 20:35:08.665580 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:08.665263    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82880fb9-9943-4d5d-b23e-b5c3c05319fc","Type":"ContainerStarted","Data":"fec338f1252229aed053558be33d306c7188413c8af2856bfa7a123c4ac9ecd2"}
Sep 09 20:35:08.894420 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:08.894385    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06acd67c-1f5b-4053-9aa9-5147c56411e4" path="/var/lib/kubelet/pods/06acd67c-1f5b-4053-9aa9-5147c56411e4/volumes"
Sep 09 20:35:09.673031 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:09.672994    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82880fb9-9943-4d5d-b23e-b5c3c05319fc","Type":"ContainerStarted","Data":"8b8c4cdff003e890a56e879b6cf56d81a60a9bbd7b9a3737a3f1d4cb10ee3e55"}
Sep 09 20:35:09.673442 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:09.673067    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82880fb9-9943-4d5d-b23e-b5c3c05319fc","Type":"ContainerStarted","Data":"252d4557b273a6d5baccbf7d4bd76027d019884e295dd807f6d8e81166665b05"}
Sep 09 20:35:09.673442 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:09.673098    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82880fb9-9943-4d5d-b23e-b5c3c05319fc","Type":"ContainerStarted","Data":"a68a4c475bbc5ceeef10be0e3265c14654cbdf68191354d7e4219c023e5a8593"}
Sep 09 20:35:09.673442 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:09.673123    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82880fb9-9943-4d5d-b23e-b5c3c05319fc","Type":"ContainerStarted","Data":"35c775cea38ca54aaf548c213447c0f12bd9a704c744ade7e35f0282485dc313"}
Sep 09 20:35:09.673442 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:09.673147    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82880fb9-9943-4d5d-b23e-b5c3c05319fc","Type":"ContainerStarted","Data":"93512ed4bec6996f5a6299bc62081bd6793ff965512c03c917f8d85fcb783a0e"}
Sep 09 20:35:09.673442 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:09.673173    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82880fb9-9943-4d5d-b23e-b5c3c05319fc","Type":"ContainerStarted","Data":"4767bcb34608b73520602d0f9bfdc36cfa86934a5007a035fbf4744aa8361d07"}
Sep 09 20:35:09.720659 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:09.720614    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.700914191 podStartE2EDuration="2.720600015s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.685824ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:35:09.720353331 +0000 UTC m=+155.400313650" watchObservedRunningTime="2026-09-09 20:35:09.720600015 +0000 UTC m=+155.400560323"
Sep 09 20:35:13.030493 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:35:13.030461    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:36:08.030554 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:36:08.030519    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:36:08.045650 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:36:08.045622    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:36:08.928827 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:36:08.928791    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:37:34.659408 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:37:34.659381    2556 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:39:36.205210 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:39:36.205177    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gk2tx"]
Sep 09 20:39:36.208288 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:39:36.208271    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-gk2tx"
Sep 09 20:39:36.211046 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:39:36.211031    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-rkxlj\""
Sep 09 20:39:36.211165 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:39:36.211140    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 20:39:36.211216 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:39:36.211195    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 20:39:36.218692 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:39:36.218672    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gk2tx"]
Sep 09 20:39:36.304508 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:39:36.304484    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkz5m\" (UniqueName: \"kubernetes.io/projected/cfb73ccb-4e37-4e6c-a6df-b4e4fc2a7d34-kube-api-access-tkz5m\") pod \"cert-manager-55b4d954d5-gk2tx\" (UID: \"cfb73ccb-4e37-4e6c-a6df-b4e4fc2a7d34\") " pod="cert-manager/cert-manager-55b4d954d5-gk2tx"
Sep 09 20:39:36.304607 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:39:36.304526    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfb73ccb-4e37-4e6c-a6df-b4e4fc2a7d34-bound-sa-token\") pod \"cert-manager-55b4d954d5-gk2tx\" (UID: \"cfb73ccb-4e37-4e6c-a6df-b4e4fc2a7d34\") " pod="cert-manager/cert-manager-55b4d954d5-gk2tx"
Sep 09 20:39:36.405701 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:39:36.405680    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkz5m\" (UniqueName: \"kubernetes.io/projected/cfb73ccb-4e37-4e6c-a6df-b4e4fc2a7d34-kube-api-access-tkz5m\") pod \"cert-manager-55b4d954d5-gk2tx\" (UID: \"cfb73ccb-4e37-4e6c-a6df-b4e4fc2a7d34\") " pod="cert-manager/cert-manager-55b4d954d5-gk2tx"
Sep 09 20:39:36.405788 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:39:36.405718    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfb73ccb-4e37-4e6c-a6df-b4e4fc2a7d34-bound-sa-token\") pod \"cert-manager-55b4d954d5-gk2tx\" (UID: \"cfb73ccb-4e37-4e6c-a6df-b4e4fc2a7d34\") " pod="cert-manager/cert-manager-55b4d954d5-gk2tx"
Sep 09 20:39:36.415482 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:39:36.415455    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfb73ccb-4e37-4e6c-a6df-b4e4fc2a7d34-bound-sa-token\") pod \"cert-manager-55b4d954d5-gk2tx\" (UID: \"cfb73ccb-4e37-4e6c-a6df-b4e4fc2a7d34\") " pod="cert-manager/cert-manager-55b4d954d5-gk2tx"
Sep 09 20:39:36.415604 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:39:36.415586    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkz5m\" (UniqueName: \"kubernetes.io/projected/cfb73ccb-4e37-4e6c-a6df-b4e4fc2a7d34-kube-api-access-tkz5m\") pod \"cert-manager-55b4d954d5-gk2tx\" (UID: \"cfb73ccb-4e37-4e6c-a6df-b4e4fc2a7d34\") " pod="cert-manager/cert-manager-55b4d954d5-gk2tx"
Sep 09 20:39:36.527323 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:39:36.527300    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-gk2tx"
Sep 09 20:39:36.802958 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:39:36.802878    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gk2tx"]
Sep 09 20:39:36.805865 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:39:36.805837    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfb73ccb_4e37_4e6c_a6df_b4e4fc2a7d34.slice/crio-1adf32b190dfb7e7d65f12d573bd4013c2805f4c96851f701f5f90a9b2a6bc35 WatchSource:0}: Error finding container 1adf32b190dfb7e7d65f12d573bd4013c2805f4c96851f701f5f90a9b2a6bc35: Status 404 returned error can't find the container with id 1adf32b190dfb7e7d65f12d573bd4013c2805f4c96851f701f5f90a9b2a6bc35
Sep 09 20:39:36.807609 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:39:36.807594    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:39:37.729191 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:39:37.729151    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-gk2tx" event={"ID":"cfb73ccb-4e37-4e6c-a6df-b4e4fc2a7d34","Type":"ContainerStarted","Data":"1adf32b190dfb7e7d65f12d573bd4013c2805f4c96851f701f5f90a9b2a6bc35"}
Sep 09 20:39:39.740416 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:39:39.740383    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-gk2tx" event={"ID":"cfb73ccb-4e37-4e6c-a6df-b4e4fc2a7d34","Type":"ContainerStarted","Data":"5b1ea0f76ca6cceba211db762b74d7bfb0a530b424060424aa6804bb9fa42996"}
Sep 09 20:39:39.758337 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:39:39.758291    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-gk2tx" podStartSLOduration=0.995353876 podStartE2EDuration="3.758276903s" totalImagesPullingTime="2.762923027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:39:39.75604916 +0000 UTC m=+425.436009480" watchObservedRunningTime="2026-09-09 20:39:39.758276903 +0000 UTC m=+425.438237211"
Sep 09 20:40:00.637314 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:00.637228    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cdf7f9947-sk8cw"]
Sep 09 20:40:00.639693 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:00.639672    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cdf7f9947-sk8cw"
Sep 09 20:40:00.644331 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:00.644305    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 20:40:00.644482 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:00.644371    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:40:00.644482 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:00.644313    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 20:40:00.644482 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:00.644313    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 20:40:00.644695 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:00.644642    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 20:40:00.644695 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:00.644649    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hbn94\""
Sep 09 20:40:00.652250 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:00.652229    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cdf7f9947-sk8cw"]
Sep 09 20:40:00.695382 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:00.695357    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/859c9262-88b7-4d3b-a392-167f260fbb8a-metrics-cert\") pod \"lws-controller-manager-6cdf7f9947-sk8cw\" (UID: \"859c9262-88b7-4d3b-a392-167f260fbb8a\") " pod="openshift-lws-operator/lws-controller-manager-6cdf7f9947-sk8cw"
Sep 09 20:40:00.695469 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:00.695387    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/859c9262-88b7-4d3b-a392-167f260fbb8a-manager-config\") pod \"lws-controller-manager-6cdf7f9947-sk8cw\" (UID: \"859c9262-88b7-4d3b-a392-167f260fbb8a\") " pod="openshift-lws-operator/lws-controller-manager-6cdf7f9947-sk8cw"
Sep 09 20:40:00.695469 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:00.695416    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk4ls\" (UniqueName: \"kubernetes.io/projected/859c9262-88b7-4d3b-a392-167f260fbb8a-kube-api-access-bk4ls\") pod \"lws-controller-manager-6cdf7f9947-sk8cw\" (UID: \"859c9262-88b7-4d3b-a392-167f260fbb8a\") " pod="openshift-lws-operator/lws-controller-manager-6cdf7f9947-sk8cw"
Sep 09 20:40:00.695549 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:00.695490    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/859c9262-88b7-4d3b-a392-167f260fbb8a-cert\") pod \"lws-controller-manager-6cdf7f9947-sk8cw\" (UID: \"859c9262-88b7-4d3b-a392-167f260fbb8a\") " pod="openshift-lws-operator/lws-controller-manager-6cdf7f9947-sk8cw"
Sep 09 20:40:00.796420 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:00.796396    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/859c9262-88b7-4d3b-a392-167f260fbb8a-metrics-cert\") pod \"lws-controller-manager-6cdf7f9947-sk8cw\" (UID: \"859c9262-88b7-4d3b-a392-167f260fbb8a\") " pod="openshift-lws-operator/lws-controller-manager-6cdf7f9947-sk8cw"
Sep 09 20:40:00.796540 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:00.796428    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/859c9262-88b7-4d3b-a392-167f260fbb8a-manager-config\") pod \"lws-controller-manager-6cdf7f9947-sk8cw\" (UID: \"859c9262-88b7-4d3b-a392-167f260fbb8a\") " pod="openshift-lws-operator/lws-controller-manager-6cdf7f9947-sk8cw"
Sep 09 20:40:00.796540 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:00.796457    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk4ls\" (UniqueName: \"kubernetes.io/projected/859c9262-88b7-4d3b-a392-167f260fbb8a-kube-api-access-bk4ls\") pod \"lws-controller-manager-6cdf7f9947-sk8cw\" (UID: \"859c9262-88b7-4d3b-a392-167f260fbb8a\") " pod="openshift-lws-operator/lws-controller-manager-6cdf7f9947-sk8cw"
Sep 09 20:40:00.796540 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:00.796494    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/859c9262-88b7-4d3b-a392-167f260fbb8a-cert\") pod \"lws-controller-manager-6cdf7f9947-sk8cw\" (UID: \"859c9262-88b7-4d3b-a392-167f260fbb8a\") " pod="openshift-lws-operator/lws-controller-manager-6cdf7f9947-sk8cw"
Sep 09 20:40:00.797152 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:00.797128    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/859c9262-88b7-4d3b-a392-167f260fbb8a-manager-config\") pod \"lws-controller-manager-6cdf7f9947-sk8cw\" (UID: \"859c9262-88b7-4d3b-a392-167f260fbb8a\") " pod="openshift-lws-operator/lws-controller-manager-6cdf7f9947-sk8cw"
Sep 09 20:40:00.798846 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:00.798827    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/859c9262-88b7-4d3b-a392-167f260fbb8a-metrics-cert\") pod \"lws-controller-manager-6cdf7f9947-sk8cw\" (UID: \"859c9262-88b7-4d3b-a392-167f260fbb8a\") " pod="openshift-lws-operator/lws-controller-manager-6cdf7f9947-sk8cw"
Sep 09 20:40:00.798946 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:00.798868    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/859c9262-88b7-4d3b-a392-167f260fbb8a-cert\") pod \"lws-controller-manager-6cdf7f9947-sk8cw\" (UID: \"859c9262-88b7-4d3b-a392-167f260fbb8a\") " pod="openshift-lws-operator/lws-controller-manager-6cdf7f9947-sk8cw"
Sep 09 20:40:00.805033 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:00.805011    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk4ls\" (UniqueName: \"kubernetes.io/projected/859c9262-88b7-4d3b-a392-167f260fbb8a-kube-api-access-bk4ls\") pod \"lws-controller-manager-6cdf7f9947-sk8cw\" (UID: \"859c9262-88b7-4d3b-a392-167f260fbb8a\") " pod="openshift-lws-operator/lws-controller-manager-6cdf7f9947-sk8cw"
Sep 09 20:40:00.948454 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:00.948392    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cdf7f9947-sk8cw"
Sep 09 20:40:01.219221 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:01.219199    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cdf7f9947-sk8cw"]
Sep 09 20:40:01.221570 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:40:01.221547    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859c9262_88b7_4d3b_a392_167f260fbb8a.slice/crio-93b1df5d130991844b2aa9f3036a31250b61de8855b4af1ae373a77a428776a6 WatchSource:0}: Error finding container 93b1df5d130991844b2aa9f3036a31250b61de8855b4af1ae373a77a428776a6: Status 404 returned error can't find the container with id 93b1df5d130991844b2aa9f3036a31250b61de8855b4af1ae373a77a428776a6
Sep 09 20:40:01.831683 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:01.831650    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cdf7f9947-sk8cw" event={"ID":"859c9262-88b7-4d3b-a392-167f260fbb8a","Type":"ContainerStarted","Data":"93b1df5d130991844b2aa9f3036a31250b61de8855b4af1ae373a77a428776a6"}
Sep 09 20:40:04.847033 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:04.846996    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cdf7f9947-sk8cw" event={"ID":"859c9262-88b7-4d3b-a392-167f260fbb8a","Type":"ContainerStarted","Data":"eee2fcc1f181ea9528982ea723d62771e431f07b15e8e65d4b6b4a570314cd19"}
Sep 09 20:40:04.847393 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:04.847126    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6cdf7f9947-sk8cw"
Sep 09 20:40:04.864970 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:04.864926    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6cdf7f9947-sk8cw" podStartSLOduration=1.8097395139999999 podStartE2EDuration="4.864893884s" totalImagesPullingTime="3.05515437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:04.863960946 +0000 UTC m=+450.543921258" watchObservedRunningTime="2026-09-09 20:40:04.864893884 +0000 UTC m=+450.544854193"
Sep 09 20:40:12.687844 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:12.687809    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h"]
Sep 09 20:40:12.690408 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:12.690392    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h"
Sep 09 20:40:12.693178 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:12.693152    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-p8mdg\""
Sep 09 20:40:12.693311 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:12.693156    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 20:40:12.693384 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:12.693319    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 20:40:12.693482 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:12.693462    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 20:40:12.693591 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:12.693508    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 20:40:12.707849 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:12.707824    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h"]
Sep 09 20:40:12.790190 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:12.790165    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50d78a07-27e0-41e1-9ecf-a9b9882eda59-tmp\") pod \"opendatahub-operator-controller-manager-859cf57674-j278h\" (UID: \"50d78a07-27e0-41e1-9ecf-a9b9882eda59\") " pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h"
Sep 09 20:40:12.790317 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:12.790196    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfkmb\" (UniqueName: \"kubernetes.io/projected/50d78a07-27e0-41e1-9ecf-a9b9882eda59-kube-api-access-kfkmb\") pod \"opendatahub-operator-controller-manager-859cf57674-j278h\" (UID: \"50d78a07-27e0-41e1-9ecf-a9b9882eda59\") " pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h"
Sep 09 20:40:12.790317 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:12.790217    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/50d78a07-27e0-41e1-9ecf-a9b9882eda59-apiservice-cert\") pod \"opendatahub-operator-controller-manager-859cf57674-j278h\" (UID: \"50d78a07-27e0-41e1-9ecf-a9b9882eda59\") " pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h"
Sep 09 20:40:12.790317 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:12.790268    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50d78a07-27e0-41e1-9ecf-a9b9882eda59-webhook-cert\") pod \"opendatahub-operator-controller-manager-859cf57674-j278h\" (UID: \"50d78a07-27e0-41e1-9ecf-a9b9882eda59\") " pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h"
Sep 09 20:40:12.790463 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:12.790322    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/50d78a07-27e0-41e1-9ecf-a9b9882eda59-manifests\") pod \"opendatahub-operator-controller-manager-859cf57674-j278h\" (UID: \"50d78a07-27e0-41e1-9ecf-a9b9882eda59\") " pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h"
Sep 09 20:40:12.890757 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:12.890731    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/50d78a07-27e0-41e1-9ecf-a9b9882eda59-manifests\") pod \"opendatahub-operator-controller-manager-859cf57674-j278h\" (UID: \"50d78a07-27e0-41e1-9ecf-a9b9882eda59\") " pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h"
Sep 09 20:40:12.890860 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:12.890831    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50d78a07-27e0-41e1-9ecf-a9b9882eda59-tmp\") pod \"opendatahub-operator-controller-manager-859cf57674-j278h\" (UID: \"50d78a07-27e0-41e1-9ecf-a9b9882eda59\") " pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h"
Sep 09 20:40:12.890901 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:12.890857    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfkmb\" (UniqueName: \"kubernetes.io/projected/50d78a07-27e0-41e1-9ecf-a9b9882eda59-kube-api-access-kfkmb\") pod \"opendatahub-operator-controller-manager-859cf57674-j278h\" (UID: \"50d78a07-27e0-41e1-9ecf-a9b9882eda59\") " pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h"
Sep 09 20:40:12.890901 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:12.890884    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/50d78a07-27e0-41e1-9ecf-a9b9882eda59-apiservice-cert\") pod \"opendatahub-operator-controller-manager-859cf57674-j278h\" (UID: \"50d78a07-27e0-41e1-9ecf-a9b9882eda59\") " pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h"
Sep 09 20:40:12.890997 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:12.890946    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50d78a07-27e0-41e1-9ecf-a9b9882eda59-webhook-cert\") pod \"opendatahub-operator-controller-manager-859cf57674-j278h\" (UID: \"50d78a07-27e0-41e1-9ecf-a9b9882eda59\") " pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h"
Sep 09 20:40:12.891160 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:12.891139    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/50d78a07-27e0-41e1-9ecf-a9b9882eda59-manifests\") pod \"opendatahub-operator-controller-manager-859cf57674-j278h\" (UID: \"50d78a07-27e0-41e1-9ecf-a9b9882eda59\") " pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h"
Sep 09 20:40:12.893104 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:12.893075    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50d78a07-27e0-41e1-9ecf-a9b9882eda59-tmp\") pod \"opendatahub-operator-controller-manager-859cf57674-j278h\" (UID: \"50d78a07-27e0-41e1-9ecf-a9b9882eda59\") " pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h"
Sep 09 20:40:12.893210 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:12.893193    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50d78a07-27e0-41e1-9ecf-a9b9882eda59-webhook-cert\") pod \"opendatahub-operator-controller-manager-859cf57674-j278h\" (UID: \"50d78a07-27e0-41e1-9ecf-a9b9882eda59\") " pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h"
Sep 09 20:40:12.893339 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:12.893324    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/50d78a07-27e0-41e1-9ecf-a9b9882eda59-apiservice-cert\") pod \"opendatahub-operator-controller-manager-859cf57674-j278h\" (UID: \"50d78a07-27e0-41e1-9ecf-a9b9882eda59\") " pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h"
Sep 09 20:40:12.899361 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:12.899339    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfkmb\" (UniqueName: \"kubernetes.io/projected/50d78a07-27e0-41e1-9ecf-a9b9882eda59-kube-api-access-kfkmb\") pod \"opendatahub-operator-controller-manager-859cf57674-j278h\" (UID: \"50d78a07-27e0-41e1-9ecf-a9b9882eda59\") " pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h"
Sep 09 20:40:13.001083 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:13.001061    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h"
Sep 09 20:40:13.276186 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:13.276117    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h"]
Sep 09 20:40:13.279561 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:40:13.279536    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d78a07_27e0_41e1_9ecf_a9b9882eda59.slice/crio-b6ed894fb8b7d608032e0bf875937c2983568a41888c1d3fa62355476bed38dc WatchSource:0}: Error finding container b6ed894fb8b7d608032e0bf875937c2983568a41888c1d3fa62355476bed38dc: Status 404 returned error can't find the container with id b6ed894fb8b7d608032e0bf875937c2983568a41888c1d3fa62355476bed38dc
Sep 09 20:40:13.885059 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:13.885016    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h" event={"ID":"50d78a07-27e0-41e1-9ecf-a9b9882eda59","Type":"ContainerStarted","Data":"b6ed894fb8b7d608032e0bf875937c2983568a41888c1d3fa62355476bed38dc"}
Sep 09 20:40:15.852875 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:15.852841    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6cdf7f9947-sk8cw"
Sep 09 20:40:16.901965 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:16.901935    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h" event={"ID":"50d78a07-27e0-41e1-9ecf-a9b9882eda59","Type":"ContainerStarted","Data":"4650dc1d055f7e6725efae4a653d5b275f1295b1fc7f4a02cf4b8693534b5fd7"}
Sep 09 20:40:17.907485 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:17.907449    2556 generic.go:356] "Generic (PLEG): container finished" podID="50d78a07-27e0-41e1-9ecf-a9b9882eda59" containerID="4650dc1d055f7e6725efae4a653d5b275f1295b1fc7f4a02cf4b8693534b5fd7" exitCode=0
Sep 09 20:40:17.907840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:17.907538    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h" event={"ID":"50d78a07-27e0-41e1-9ecf-a9b9882eda59","Type":"ContainerDied","Data":"4650dc1d055f7e6725efae4a653d5b275f1295b1fc7f4a02cf4b8693534b5fd7"}
Sep 09 20:40:18.913560 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:18.913520    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h" event={"ID":"50d78a07-27e0-41e1-9ecf-a9b9882eda59","Type":"ContainerStarted","Data":"ce043ec48494c2d7501d3ac6904b946bdacc115514815fba93f1648705227664"}
Sep 09 20:40:18.913971 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:18.913616    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h"
Sep 09 20:40:29.920924 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:29.920873    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h"
Sep 09 20:40:29.944309 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:29.944246    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-859cf57674-j278h" podStartSLOduration=13.496352773 podStartE2EDuration="17.944229478s" totalImagesPullingTime="3.370150956s" totalInitContainerRuntime="1.077725749s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:40:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:18.939348201 +0000 UTC m=+464.619308509" watchObservedRunningTime="2026-09-09 20:40:29.944229478 +0000 UTC m=+475.624189789"
Sep 09 20:40:32.094436 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:32.094403    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-65d657c8d5-ptlv5"]
Sep 09 20:40:32.106940 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:32.106917    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65d657c8d5-ptlv5"
Sep 09 20:40:32.110256 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:32.110232    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 20:40:32.111338 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:32.111319    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 20:40:32.111440 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:32.111422    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 20:40:32.111489 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:32.111474    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pn5gg\""
Sep 09 20:40:32.111526 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:32.111515    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 20:40:32.137334 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:32.137311    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65d657c8d5-ptlv5"]
Sep 09 20:40:32.257018 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:32.256992    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzd5r\" (UniqueName: \"kubernetes.io/projected/b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d-kube-api-access-zzd5r\") pod \"kube-auth-proxy-65d657c8d5-ptlv5\" (UID: \"b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d\") " pod="openshift-ingress/kube-auth-proxy-65d657c8d5-ptlv5"
Sep 09 20:40:32.257128 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:32.257049    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d-tmp\") pod \"kube-auth-proxy-65d657c8d5-ptlv5\" (UID: \"b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d\") " pod="openshift-ingress/kube-auth-proxy-65d657c8d5-ptlv5"
Sep 09 20:40:32.257193 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:32.257144    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d-tls-certs\") pod \"kube-auth-proxy-65d657c8d5-ptlv5\" (UID: \"b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d\") " pod="openshift-ingress/kube-auth-proxy-65d657c8d5-ptlv5"
Sep 09 20:40:32.357708 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:32.357648    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d-tls-certs\") pod \"kube-auth-proxy-65d657c8d5-ptlv5\" (UID: \"b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d\") " pod="openshift-ingress/kube-auth-proxy-65d657c8d5-ptlv5"
Sep 09 20:40:32.357708 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:32.357681    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzd5r\" (UniqueName: \"kubernetes.io/projected/b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d-kube-api-access-zzd5r\") pod \"kube-auth-proxy-65d657c8d5-ptlv5\" (UID: \"b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d\") " pod="openshift-ingress/kube-auth-proxy-65d657c8d5-ptlv5"
Sep 09 20:40:32.357870 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:32.357725    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d-tmp\") pod \"kube-auth-proxy-65d657c8d5-ptlv5\" (UID: \"b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d\") " pod="openshift-ingress/kube-auth-proxy-65d657c8d5-ptlv5"
Sep 09 20:40:32.359953 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:32.359933    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d-tmp\") pod \"kube-auth-proxy-65d657c8d5-ptlv5\" (UID: \"b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d\") " pod="openshift-ingress/kube-auth-proxy-65d657c8d5-ptlv5"
Sep 09 20:40:32.360275 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:32.360254    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d-tls-certs\") pod \"kube-auth-proxy-65d657c8d5-ptlv5\" (UID: \"b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d\") " pod="openshift-ingress/kube-auth-proxy-65d657c8d5-ptlv5"
Sep 09 20:40:32.368760 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:32.368735    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzd5r\" (UniqueName: \"kubernetes.io/projected/b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d-kube-api-access-zzd5r\") pod \"kube-auth-proxy-65d657c8d5-ptlv5\" (UID: \"b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d\") " pod="openshift-ingress/kube-auth-proxy-65d657c8d5-ptlv5"
Sep 09 20:40:32.416356 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:32.416336    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65d657c8d5-ptlv5"
Sep 09 20:40:32.728673 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:32.728601    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65d657c8d5-ptlv5"]
Sep 09 20:40:32.732344 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:40:32.732321    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b0a5e1_d4b8_4dcc_ac2a_57f1ccee756d.slice/crio-afaf475173e0494aac3ba834bb934450f622659a8c05845ef47137a9481b5c32 WatchSource:0}: Error finding container afaf475173e0494aac3ba834bb934450f622659a8c05845ef47137a9481b5c32: Status 404 returned error can't find the container with id afaf475173e0494aac3ba834bb934450f622659a8c05845ef47137a9481b5c32
Sep 09 20:40:32.971610 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:32.971578    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65d657c8d5-ptlv5" event={"ID":"b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d","Type":"ContainerStarted","Data":"afaf475173e0494aac3ba834bb934450f622659a8c05845ef47137a9481b5c32"}
Sep 09 20:40:35.442760 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:35.442738    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 20:40:35.988533 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:35.988497    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65d657c8d5-ptlv5" event={"ID":"b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d","Type":"ContainerStarted","Data":"749d84b9dd5b94d32b1654cf66f885f9ed8a599757190faf87e7a3320e3b867f"}
Sep 09 20:40:36.006245 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:40:36.006198    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-65d657c8d5-ptlv5" podStartSLOduration=1.300183423 podStartE2EDuration="4.006182677s" totalImagesPullingTime="2.705999254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:36.004866972 +0000 UTC m=+481.684827283" watchObservedRunningTime="2026-09-09 20:40:36.006182677 +0000 UTC m=+481.686142986"
Sep 09 20:42:15.136052 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:42:15.136022    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-zn5ms"]
Sep 09 20:42:15.138393 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:42:15.138372    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zn5ms"
Sep 09 20:42:15.142187 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:42:15.142163    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 09 20:42:15.142404 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:42:15.142389    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 09 20:42:15.142478 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:42:15.142459    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 20:42:15.143537 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:42:15.143517    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nm6fs\""
Sep 09 20:42:15.143650 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:42:15.143582    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 20:42:15.148725 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:42:15.148706    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-zn5ms"]
Sep 09 20:42:15.249296 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:42:15.249271    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ce3f6259-dd9e-4305-9348-472835f0f076-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-zn5ms\" (UID: \"ce3f6259-dd9e-4305-9348-472835f0f076\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zn5ms"
Sep 09 20:42:15.249402 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:42:15.249384    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lkht\" (UniqueName: \"kubernetes.io/projected/ce3f6259-dd9e-4305-9348-472835f0f076-kube-api-access-8lkht\") pod \"kuadrant-console-plugin-fbdd4f445-zn5ms\" (UID: \"ce3f6259-dd9e-4305-9348-472835f0f076\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zn5ms"
Sep 09 20:42:15.249446 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:42:15.249427    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce3f6259-dd9e-4305-9348-472835f0f076-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-zn5ms\" (UID: \"ce3f6259-dd9e-4305-9348-472835f0f076\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zn5ms"
Sep 09 20:42:15.350007 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:42:15.349979    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lkht\" (UniqueName: \"kubernetes.io/projected/ce3f6259-dd9e-4305-9348-472835f0f076-kube-api-access-8lkht\") pod \"kuadrant-console-plugin-fbdd4f445-zn5ms\" (UID: \"ce3f6259-dd9e-4305-9348-472835f0f076\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zn5ms"
Sep 09 20:42:15.350145 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:42:15.350024    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce3f6259-dd9e-4305-9348-472835f0f076-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-zn5ms\" (UID: \"ce3f6259-dd9e-4305-9348-472835f0f076\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zn5ms"
Sep 09 20:42:15.350145 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:42:15.350057    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ce3f6259-dd9e-4305-9348-472835f0f076-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-zn5ms\" (UID: \"ce3f6259-dd9e-4305-9348-472835f0f076\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zn5ms"
Sep 09 20:42:15.350267 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:42:15.350169    2556 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 09 20:42:15.350267 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:42:15.350251    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce3f6259-dd9e-4305-9348-472835f0f076-plugin-serving-cert podName:ce3f6259-dd9e-4305-9348-472835f0f076 nodeName:}" failed. No retries permitted until 2026-09-09 20:42:15.850228937 +0000 UTC m=+581.530189248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/ce3f6259-dd9e-4305-9348-472835f0f076-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-zn5ms" (UID: "ce3f6259-dd9e-4305-9348-472835f0f076") : secret "plugin-serving-cert" not found
Sep 09 20:42:15.350673 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:42:15.350655    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ce3f6259-dd9e-4305-9348-472835f0f076-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-zn5ms\" (UID: \"ce3f6259-dd9e-4305-9348-472835f0f076\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zn5ms"
Sep 09 20:42:15.359525 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:42:15.359499    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lkht\" (UniqueName: \"kubernetes.io/projected/ce3f6259-dd9e-4305-9348-472835f0f076-kube-api-access-8lkht\") pod \"kuadrant-console-plugin-fbdd4f445-zn5ms\" (UID: \"ce3f6259-dd9e-4305-9348-472835f0f076\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zn5ms"
Sep 09 20:42:15.854218 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:42:15.854187    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce3f6259-dd9e-4305-9348-472835f0f076-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-zn5ms\" (UID: \"ce3f6259-dd9e-4305-9348-472835f0f076\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zn5ms"
Sep 09 20:42:15.856469 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:42:15.856440    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce3f6259-dd9e-4305-9348-472835f0f076-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-zn5ms\" (UID: \"ce3f6259-dd9e-4305-9348-472835f0f076\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zn5ms"
Sep 09 20:42:16.048667 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:42:16.048635    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zn5ms"
Sep 09 20:42:16.321216 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:42:16.321183    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-zn5ms"]
Sep 09 20:42:16.324522 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:42:16.324494    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce3f6259_dd9e_4305_9348_472835f0f076.slice/crio-2f2f2097c12889977fae52abc170ac0a411273c165eea5ff1b8ad1b0994e8530 WatchSource:0}: Error finding container 2f2f2097c12889977fae52abc170ac0a411273c165eea5ff1b8ad1b0994e8530: Status 404 returned error can't find the container with id 2f2f2097c12889977fae52abc170ac0a411273c165eea5ff1b8ad1b0994e8530
Sep 09 20:42:16.436333 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:42:16.436295    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zn5ms" event={"ID":"ce3f6259-dd9e-4305-9348-472835f0f076","Type":"ContainerStarted","Data":"2f2f2097c12889977fae52abc170ac0a411273c165eea5ff1b8ad1b0994e8530"}
Sep 09 20:42:22.470612 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:42:22.470533    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zn5ms" event={"ID":"ce3f6259-dd9e-4305-9348-472835f0f076","Type":"ContainerStarted","Data":"c14d5c67abf0d57f1907c29517b4a042aca30d10a28f151a49c4afde916da56a"}
Sep 09 20:42:22.490751 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:42:22.490707    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zn5ms" podStartSLOduration=1.615162764 podStartE2EDuration="7.490693183s" totalImagesPullingTime="5.875530419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:42:22.488241184 +0000 UTC m=+588.168201493" watchObservedRunningTime="2026-09-09 20:42:22.490693183 +0000 UTC m=+588.170653493"
Sep 09 20:43:08.604868 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:43:08.604786    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:43:08.608257 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:43:08.608240    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-tpptr"
Sep 09 20:43:08.610777 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:43:08.610755    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 20:43:08.617394 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:43:08.617370    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:43:08.642457 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:43:08.642435    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:43:08.754315 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:43:08.754287    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d63afcf8-a51a-48cb-a24f-4a222cbf2600-config-file\") pod \"limitador-limitador-84994f7ffc-tpptr\" (UID: \"d63afcf8-a51a-48cb-a24f-4a222cbf2600\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tpptr"
Sep 09 20:43:08.754422 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:43:08.754350    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b62ng\" (UniqueName: \"kubernetes.io/projected/d63afcf8-a51a-48cb-a24f-4a222cbf2600-kube-api-access-b62ng\") pod \"limitador-limitador-84994f7ffc-tpptr\" (UID: \"d63afcf8-a51a-48cb-a24f-4a222cbf2600\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tpptr"
Sep 09 20:43:08.854838 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:43:08.854812    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d63afcf8-a51a-48cb-a24f-4a222cbf2600-config-file\") pod \"limitador-limitador-84994f7ffc-tpptr\" (UID: \"d63afcf8-a51a-48cb-a24f-4a222cbf2600\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tpptr"
Sep 09 20:43:08.854984 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:43:08.854862    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b62ng\" (UniqueName: \"kubernetes.io/projected/d63afcf8-a51a-48cb-a24f-4a222cbf2600-kube-api-access-b62ng\") pod \"limitador-limitador-84994f7ffc-tpptr\" (UID: \"d63afcf8-a51a-48cb-a24f-4a222cbf2600\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tpptr"
Sep 09 20:43:08.855428 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:43:08.855409    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d63afcf8-a51a-48cb-a24f-4a222cbf2600-config-file\") pod \"limitador-limitador-84994f7ffc-tpptr\" (UID: \"d63afcf8-a51a-48cb-a24f-4a222cbf2600\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tpptr"
Sep 09 20:43:08.863513 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:43:08.863487    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b62ng\" (UniqueName: \"kubernetes.io/projected/d63afcf8-a51a-48cb-a24f-4a222cbf2600-kube-api-access-b62ng\") pod \"limitador-limitador-84994f7ffc-tpptr\" (UID: \"d63afcf8-a51a-48cb-a24f-4a222cbf2600\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tpptr"
Sep 09 20:43:08.918706 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:43:08.918683    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-tpptr"
Sep 09 20:43:09.188973 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:43:09.188885    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:43:09.191526 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:43:09.191500    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63afcf8_a51a_48cb_a24f_4a222cbf2600.slice/crio-afc8e548fb3c8ba4eacadec883d8d7af045412cef5afd54867c0028131b94d3e WatchSource:0}: Error finding container afc8e548fb3c8ba4eacadec883d8d7af045412cef5afd54867c0028131b94d3e: Status 404 returned error can't find the container with id afc8e548fb3c8ba4eacadec883d8d7af045412cef5afd54867c0028131b94d3e
Sep 09 20:43:09.678447 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:43:09.678417    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-tpptr" event={"ID":"d63afcf8-a51a-48cb-a24f-4a222cbf2600","Type":"ContainerStarted","Data":"afc8e548fb3c8ba4eacadec883d8d7af045412cef5afd54867c0028131b94d3e"}
Sep 09 20:43:11.688821 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:43:11.688785    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-tpptr" event={"ID":"d63afcf8-a51a-48cb-a24f-4a222cbf2600","Type":"ContainerStarted","Data":"769ef33046208b4a7a106f39162f26444a9e102a9994edc80d231193753e3c59"}
Sep 09 20:43:11.689167 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:43:11.688900    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-tpptr"
Sep 09 20:43:11.707241 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:43:11.707196    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-tpptr" podStartSLOduration=1.382403025 podStartE2EDuration="3.707181861s" totalImagesPullingTime="2.324778836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:43:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:43:11.705709117 +0000 UTC m=+637.385669434" watchObservedRunningTime="2026-09-09 20:43:11.707181861 +0000 UTC m=+637.387142170"
Sep 09 20:43:22.694113 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:43:22.694079    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-tpptr"
Sep 09 20:44:24.937560 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:24.937527    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:44:47.125341 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:47.125303    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:44:53.593185 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.593146    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"]
Sep 09 20:44:53.596816 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.596788    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:44:53.599649 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.599625    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 20:44:53.601044 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.601017    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ddfsp\""
Sep 09 20:44:53.601153 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.601041    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 09 20:44:53.601153 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.601056    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 20:44:53.607383 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.607361    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"]
Sep 09 20:44:53.659724 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.659704    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09f27f65-11bd-43aa-94fa-2a903b79be77-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9kn2j\" (UID: \"09f27f65-11bd-43aa-94fa-2a903b79be77\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:44:53.659864 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.659736    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09f27f65-11bd-43aa-94fa-2a903b79be77-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9kn2j\" (UID: \"09f27f65-11bd-43aa-94fa-2a903b79be77\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:44:53.659864 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.659773    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2wdf\" (UniqueName: \"kubernetes.io/projected/09f27f65-11bd-43aa-94fa-2a903b79be77-kube-api-access-z2wdf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9kn2j\" (UID: \"09f27f65-11bd-43aa-94fa-2a903b79be77\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:44:53.659864 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.659850    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09f27f65-11bd-43aa-94fa-2a903b79be77-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9kn2j\" (UID: \"09f27f65-11bd-43aa-94fa-2a903b79be77\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:44:53.660074 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.659885    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09f27f65-11bd-43aa-94fa-2a903b79be77-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9kn2j\" (UID: \"09f27f65-11bd-43aa-94fa-2a903b79be77\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:44:53.660074 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.660000    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f27f65-11bd-43aa-94fa-2a903b79be77-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9kn2j\" (UID: \"09f27f65-11bd-43aa-94fa-2a903b79be77\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:44:53.761048 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.761022    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09f27f65-11bd-43aa-94fa-2a903b79be77-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9kn2j\" (UID: \"09f27f65-11bd-43aa-94fa-2a903b79be77\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:44:53.761143 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.761070    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2wdf\" (UniqueName: \"kubernetes.io/projected/09f27f65-11bd-43aa-94fa-2a903b79be77-kube-api-access-z2wdf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9kn2j\" (UID: \"09f27f65-11bd-43aa-94fa-2a903b79be77\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:44:53.761143 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.761089    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09f27f65-11bd-43aa-94fa-2a903b79be77-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9kn2j\" (UID: \"09f27f65-11bd-43aa-94fa-2a903b79be77\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:44:53.761143 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.761105    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09f27f65-11bd-43aa-94fa-2a903b79be77-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9kn2j\" (UID: \"09f27f65-11bd-43aa-94fa-2a903b79be77\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:44:53.761294 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.761156    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f27f65-11bd-43aa-94fa-2a903b79be77-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9kn2j\" (UID: \"09f27f65-11bd-43aa-94fa-2a903b79be77\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:44:53.761294 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.761185    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09f27f65-11bd-43aa-94fa-2a903b79be77-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9kn2j\" (UID: \"09f27f65-11bd-43aa-94fa-2a903b79be77\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:44:53.761428 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.761410    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09f27f65-11bd-43aa-94fa-2a903b79be77-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9kn2j\" (UID: \"09f27f65-11bd-43aa-94fa-2a903b79be77\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:44:53.761505 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.761487    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f27f65-11bd-43aa-94fa-2a903b79be77-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9kn2j\" (UID: \"09f27f65-11bd-43aa-94fa-2a903b79be77\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:44:53.761568 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.761553    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09f27f65-11bd-43aa-94fa-2a903b79be77-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9kn2j\" (UID: \"09f27f65-11bd-43aa-94fa-2a903b79be77\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:44:53.763317 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.763297    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09f27f65-11bd-43aa-94fa-2a903b79be77-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9kn2j\" (UID: \"09f27f65-11bd-43aa-94fa-2a903b79be77\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:44:53.763579 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.763560    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09f27f65-11bd-43aa-94fa-2a903b79be77-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9kn2j\" (UID: \"09f27f65-11bd-43aa-94fa-2a903b79be77\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:44:53.769573 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.769556    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2wdf\" (UniqueName: \"kubernetes.io/projected/09f27f65-11bd-43aa-94fa-2a903b79be77-kube-api-access-z2wdf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9kn2j\" (UID: \"09f27f65-11bd-43aa-94fa-2a903b79be77\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:44:53.908528 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:53.908473    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:44:54.178272 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:54.178206    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"]
Sep 09 20:44:54.181645 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:44:54.181613    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f27f65_11bd_43aa_94fa_2a903b79be77.slice/crio-d466a39ee4a1c53ea65c1a91cf2f92544064256cbf4c3e07465b8130a69cd0f7 WatchSource:0}: Error finding container d466a39ee4a1c53ea65c1a91cf2f92544064256cbf4c3e07465b8130a69cd0f7: Status 404 returned error can't find the container with id d466a39ee4a1c53ea65c1a91cf2f92544064256cbf4c3e07465b8130a69cd0f7
Sep 09 20:44:54.182980 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:54.182961    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:44:54.206160 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:54.206135    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:44:55.152780 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:55.152745    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j" event={"ID":"09f27f65-11bd-43aa-94fa-2a903b79be77","Type":"ContainerStarted","Data":"d466a39ee4a1c53ea65c1a91cf2f92544064256cbf4c3e07465b8130a69cd0f7"}
Sep 09 20:44:56.159325 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:56.159284    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j" event={"ID":"09f27f65-11bd-43aa-94fa-2a903b79be77","Type":"ContainerStarted","Data":"98d6eeebf8953102944d15ec0562c67f37cc43eb46e2704b63ec77fd6dde8515"}
Sep 09 20:44:56.177573 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:44:56.177523    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j" podStartSLOduration=2.158118699 podStartE2EDuration="3.177508286s" totalImagesPullingTime="1.019389587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:44:56.175435931 +0000 UTC m=+741.855396240" watchObservedRunningTime="2026-09-09 20:44:56.177508286 +0000 UTC m=+741.857468596"
Sep 09 20:45:00.136042 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:00.136008    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816445-jxdd9"]
Sep 09 20:45:00.139434 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:00.139412    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816445-jxdd9"
Sep 09 20:45:00.142514 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:00.142493    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsks9\""
Sep 09 20:45:00.148738 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:00.148713    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816445-jxdd9"]
Sep 09 20:45:00.149629 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:00.149613    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816445-jxdd9"
Sep 09 20:45:00.417432 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:00.417366    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816445-jxdd9"]
Sep 09 20:45:00.419711 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:45:00.419684    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4caea27e_6bc7_4f35_a4da_5dd650051fee.slice/crio-7c8b03943f369084a34d3ff6508833ac9aa7256d1ce157707d19b5f314c354ef WatchSource:0}: Error finding container 7c8b03943f369084a34d3ff6508833ac9aa7256d1ce157707d19b5f314c354ef: Status 404 returned error can't find the container with id 7c8b03943f369084a34d3ff6508833ac9aa7256d1ce157707d19b5f314c354ef
Sep 09 20:45:01.183070 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:01.183030    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816445-jxdd9" event={"ID":"4caea27e-6bc7-4f35-a4da-5dd650051fee","Type":"ContainerStarted","Data":"7c8b03943f369084a34d3ff6508833ac9aa7256d1ce157707d19b5f314c354ef"}
Sep 09 20:45:03.909293 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:03.909265    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:45:03.909614 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:03.909358    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:45:03.921717 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:03.921699    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:45:04.198422 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:04.198355    2556 generic.go:356] "Generic (PLEG): container finished" podID="4caea27e-6bc7-4f35-a4da-5dd650051fee" containerID="9b5781a8248798fd06e33b385e3502841b539130c54d80f2bdc349cdd868761c" exitCode=0
Sep 09 20:45:04.198422 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:04.198408    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816445-jxdd9" event={"ID":"4caea27e-6bc7-4f35-a4da-5dd650051fee","Type":"ContainerDied","Data":"9b5781a8248798fd06e33b385e3502841b539130c54d80f2bdc349cdd868761c"}
Sep 09 20:45:04.209500 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:04.209471    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9kn2j"
Sep 09 20:45:05.322912 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:05.322888    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816445-jxdd9"
Sep 09 20:45:06.208181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:06.208151    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816445-jxdd9"
Sep 09 20:45:06.208325 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:06.208182    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816445-jxdd9" event={"ID":"4caea27e-6bc7-4f35-a4da-5dd650051fee","Type":"ContainerDied","Data":"7c8b03943f369084a34d3ff6508833ac9aa7256d1ce157707d19b5f314c354ef"}
Sep 09 20:45:06.208325 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:06.208226    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c8b03943f369084a34d3ff6508833ac9aa7256d1ce157707d19b5f314c354ef"
Sep 09 20:45:12.594117 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.594080    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"]
Sep 09 20:45:12.594472 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.594442    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4caea27e-6bc7-4f35-a4da-5dd650051fee" containerName="cleanup"
Sep 09 20:45:12.594472 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.594454    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4caea27e-6bc7-4f35-a4da-5dd650051fee" containerName="cleanup"
Sep 09 20:45:12.594541 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.594527    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4caea27e-6bc7-4f35-a4da-5dd650051fee" containerName="cleanup"
Sep 09 20:45:12.597630 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.597608    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:12.600531 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.600514    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 09 20:45:12.611889 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.611866    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"]
Sep 09 20:45:12.713594 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.713559    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3db7f59-5eea-4681-9d55-da11e9093913-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz\" (UID: \"f3db7f59-5eea-4681-9d55-da11e9093913\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:12.713721 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.713611    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3db7f59-5eea-4681-9d55-da11e9093913-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz\" (UID: \"f3db7f59-5eea-4681-9d55-da11e9093913\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:12.713721 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.713638    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3db7f59-5eea-4681-9d55-da11e9093913-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz\" (UID: \"f3db7f59-5eea-4681-9d55-da11e9093913\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:12.713721 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.713700    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp6pk\" (UniqueName: \"kubernetes.io/projected/f3db7f59-5eea-4681-9d55-da11e9093913-kube-api-access-gp6pk\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz\" (UID: \"f3db7f59-5eea-4681-9d55-da11e9093913\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:12.713721 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.713717    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3db7f59-5eea-4681-9d55-da11e9093913-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz\" (UID: \"f3db7f59-5eea-4681-9d55-da11e9093913\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:12.713884 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.713734    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3db7f59-5eea-4681-9d55-da11e9093913-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz\" (UID: \"f3db7f59-5eea-4681-9d55-da11e9093913\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:12.814846 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.814809    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3db7f59-5eea-4681-9d55-da11e9093913-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz\" (UID: \"f3db7f59-5eea-4681-9d55-da11e9093913\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:12.814968 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.814877    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp6pk\" (UniqueName: \"kubernetes.io/projected/f3db7f59-5eea-4681-9d55-da11e9093913-kube-api-access-gp6pk\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz\" (UID: \"f3db7f59-5eea-4681-9d55-da11e9093913\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:12.814968 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.814922    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3db7f59-5eea-4681-9d55-da11e9093913-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz\" (UID: \"f3db7f59-5eea-4681-9d55-da11e9093913\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:12.814968 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.814950    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3db7f59-5eea-4681-9d55-da11e9093913-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz\" (UID: \"f3db7f59-5eea-4681-9d55-da11e9093913\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:12.815145 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.815005    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3db7f59-5eea-4681-9d55-da11e9093913-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz\" (UID: \"f3db7f59-5eea-4681-9d55-da11e9093913\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:12.815145 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.815050    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3db7f59-5eea-4681-9d55-da11e9093913-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz\" (UID: \"f3db7f59-5eea-4681-9d55-da11e9093913\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:12.815269 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.815250    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3db7f59-5eea-4681-9d55-da11e9093913-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz\" (UID: \"f3db7f59-5eea-4681-9d55-da11e9093913\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:12.815332 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.815317    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3db7f59-5eea-4681-9d55-da11e9093913-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz\" (UID: \"f3db7f59-5eea-4681-9d55-da11e9093913\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:12.815393 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.815365    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3db7f59-5eea-4681-9d55-da11e9093913-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz\" (UID: \"f3db7f59-5eea-4681-9d55-da11e9093913\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:12.817057 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.817037    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3db7f59-5eea-4681-9d55-da11e9093913-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz\" (UID: \"f3db7f59-5eea-4681-9d55-da11e9093913\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:12.817355 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.817338    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3db7f59-5eea-4681-9d55-da11e9093913-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz\" (UID: \"f3db7f59-5eea-4681-9d55-da11e9093913\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:12.824089 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.824069    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp6pk\" (UniqueName: \"kubernetes.io/projected/f3db7f59-5eea-4681-9d55-da11e9093913-kube-api-access-gp6pk\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz\" (UID: \"f3db7f59-5eea-4681-9d55-da11e9093913\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:12.907679 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:12.907626    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:13.176574 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:13.176501    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"]
Sep 09 20:45:13.178859 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:45:13.178830    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3db7f59_5eea_4681_9d55_da11e9093913.slice/crio-15e8269531f7e2f2431b2773915603951c92200eaf579eaa483052f7adcb02da WatchSource:0}: Error finding container 15e8269531f7e2f2431b2773915603951c92200eaf579eaa483052f7adcb02da: Status 404 returned error can't find the container with id 15e8269531f7e2f2431b2773915603951c92200eaf579eaa483052f7adcb02da
Sep 09 20:45:13.238395 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:13.238369    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz" event={"ID":"f3db7f59-5eea-4681-9d55-da11e9093913","Type":"ContainerStarted","Data":"15e8269531f7e2f2431b2773915603951c92200eaf579eaa483052f7adcb02da"}
Sep 09 20:45:13.492856 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:13.492827    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:45:14.244819 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:14.244781    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz" event={"ID":"f3db7f59-5eea-4681-9d55-da11e9093913","Type":"ContainerStarted","Data":"648751e93a97b0b00f54e1c105bac82395747cd1a3b8ab51e3a15f84beb20e4a"}
Sep 09 20:45:14.265351 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:14.265302    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz" podStartSLOduration=2.022753859 podStartE2EDuration="2.265287284s" totalImagesPullingTime="242.533425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:45:14.263833273 +0000 UTC m=+759.943793582" watchObservedRunningTime="2026-09-09 20:45:14.265287284 +0000 UTC m=+759.945247592"
Sep 09 20:45:22.094274 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:22.094236    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:45:22.908635 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:22.908605    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:22.908793 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:22.908654    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:22.920949 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:22.920922    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:23.297698 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:23.297672    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz"
Sep 09 20:45:30.291720 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:30.291684    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:45:43.599957 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:43.599919    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:45:49.688456 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:49.688413    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"]
Sep 09 20:45:49.696832 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:49.696807    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:45:49.699733 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:49.699707    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 09 20:45:49.700591 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:49.700565    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"]
Sep 09 20:45:49.715087 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:49.715051    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9176bbb6-3d05-4262-afee-1ac4717ce6c2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc\" (UID: \"9176bbb6-3d05-4262-afee-1ac4717ce6c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:45:49.715188 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:49.715114    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjm9q\" (UniqueName: \"kubernetes.io/projected/9176bbb6-3d05-4262-afee-1ac4717ce6c2-kube-api-access-wjm9q\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc\" (UID: \"9176bbb6-3d05-4262-afee-1ac4717ce6c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:45:49.715188 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:49.715173    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9176bbb6-3d05-4262-afee-1ac4717ce6c2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc\" (UID: \"9176bbb6-3d05-4262-afee-1ac4717ce6c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:45:49.715304 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:49.715252    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9176bbb6-3d05-4262-afee-1ac4717ce6c2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc\" (UID: \"9176bbb6-3d05-4262-afee-1ac4717ce6c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:45:49.715358 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:49.715302    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9176bbb6-3d05-4262-afee-1ac4717ce6c2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc\" (UID: \"9176bbb6-3d05-4262-afee-1ac4717ce6c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:45:49.715400 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:49.715390    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9176bbb6-3d05-4262-afee-1ac4717ce6c2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc\" (UID: \"9176bbb6-3d05-4262-afee-1ac4717ce6c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:45:49.816636 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:49.816608    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjm9q\" (UniqueName: \"kubernetes.io/projected/9176bbb6-3d05-4262-afee-1ac4717ce6c2-kube-api-access-wjm9q\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc\" (UID: \"9176bbb6-3d05-4262-afee-1ac4717ce6c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:45:49.816746 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:49.816645    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9176bbb6-3d05-4262-afee-1ac4717ce6c2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc\" (UID: \"9176bbb6-3d05-4262-afee-1ac4717ce6c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:45:49.816746 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:49.816666    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9176bbb6-3d05-4262-afee-1ac4717ce6c2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc\" (UID: \"9176bbb6-3d05-4262-afee-1ac4717ce6c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:45:49.816746 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:49.816693    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9176bbb6-3d05-4262-afee-1ac4717ce6c2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc\" (UID: \"9176bbb6-3d05-4262-afee-1ac4717ce6c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:45:49.817003 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:49.816766    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9176bbb6-3d05-4262-afee-1ac4717ce6c2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc\" (UID: \"9176bbb6-3d05-4262-afee-1ac4717ce6c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:45:49.817003 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:49.816818    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9176bbb6-3d05-4262-afee-1ac4717ce6c2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc\" (UID: \"9176bbb6-3d05-4262-afee-1ac4717ce6c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:45:49.817159 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:49.817136    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9176bbb6-3d05-4262-afee-1ac4717ce6c2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc\" (UID: \"9176bbb6-3d05-4262-afee-1ac4717ce6c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:45:49.817240 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:49.817193    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9176bbb6-3d05-4262-afee-1ac4717ce6c2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc\" (UID: \"9176bbb6-3d05-4262-afee-1ac4717ce6c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:45:49.817240 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:49.817203    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9176bbb6-3d05-4262-afee-1ac4717ce6c2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc\" (UID: \"9176bbb6-3d05-4262-afee-1ac4717ce6c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:45:49.818963 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:49.818941    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9176bbb6-3d05-4262-afee-1ac4717ce6c2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc\" (UID: \"9176bbb6-3d05-4262-afee-1ac4717ce6c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:45:49.819204 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:49.819187    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9176bbb6-3d05-4262-afee-1ac4717ce6c2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc\" (UID: \"9176bbb6-3d05-4262-afee-1ac4717ce6c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:45:49.824017 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:49.823997    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjm9q\" (UniqueName: \"kubernetes.io/projected/9176bbb6-3d05-4262-afee-1ac4717ce6c2-kube-api-access-wjm9q\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc\" (UID: \"9176bbb6-3d05-4262-afee-1ac4717ce6c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:45:50.007490 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:50.007468    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:45:50.286927 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:50.286390    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"]
Sep 09 20:45:50.289846 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:45:50.289818    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9176bbb6_3d05_4262_afee_1ac4717ce6c2.slice/crio-578f9b39cdd3c00da88668c1d758e2407cb32bb7a1f0a96b6aae4f76a95a8e1d WatchSource:0}: Error finding container 578f9b39cdd3c00da88668c1d758e2407cb32bb7a1f0a96b6aae4f76a95a8e1d: Status 404 returned error can't find the container with id 578f9b39cdd3c00da88668c1d758e2407cb32bb7a1f0a96b6aae4f76a95a8e1d
Sep 09 20:45:50.409695 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:50.409661    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc" event={"ID":"9176bbb6-3d05-4262-afee-1ac4717ce6c2","Type":"ContainerStarted","Data":"578f9b39cdd3c00da88668c1d758e2407cb32bb7a1f0a96b6aae4f76a95a8e1d"}
Sep 09 20:45:51.415315 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:51.415282    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc" event={"ID":"9176bbb6-3d05-4262-afee-1ac4717ce6c2","Type":"ContainerStarted","Data":"3f63a6b125bd2a62d8c7b33122373a8cec94beb3fe73b8ac1998ec2ac3cf7d50"}
Sep 09 20:45:51.433590 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:51.433546    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc" podStartSLOduration=2.191565385 podStartE2EDuration="2.433532608s" totalImagesPullingTime="241.967223ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:45:51.431775948 +0000 UTC m=+797.111736312" watchObservedRunningTime="2026-09-09 20:45:51.433532608 +0000 UTC m=+797.113492949"
Sep 09 20:45:51.796664 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:45:51.796629    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:46:00.007952 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:00.007857    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:46:00.007952 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:00.007921    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:46:00.020237 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:00.020207    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:46:00.468544 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:00.468476    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc"
Sep 09 20:46:41.917374 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:41.917343    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7684-140131340642352"]
Sep 09 20:46:41.920856 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:41.920837    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140131340642352"
Sep 09 20:46:41.941893 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:41.941855    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-140131340642352"]
Sep 09 20:46:41.949569 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:41.949539    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-744w4\""
Sep 09 20:46:42.034538 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:42.034514    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z2z7\" (UniqueName: \"kubernetes.io/projected/26589e33-9304-4fa3-b9cc-2a511c09c34c-kube-api-access-2z2z7\") pod \"test-curl-7684-140131340642352\" (UID: \"26589e33-9304-4fa3-b9cc-2a511c09c34c\") " pod="opendatahub/test-curl-7684-140131340642352"
Sep 09 20:46:42.134995 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:42.134972    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z2z7\" (UniqueName: \"kubernetes.io/projected/26589e33-9304-4fa3-b9cc-2a511c09c34c-kube-api-access-2z2z7\") pod \"test-curl-7684-140131340642352\" (UID: \"26589e33-9304-4fa3-b9cc-2a511c09c34c\") " pod="opendatahub/test-curl-7684-140131340642352"
Sep 09 20:46:42.145579 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:42.145550    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z2z7\" (UniqueName: \"kubernetes.io/projected/26589e33-9304-4fa3-b9cc-2a511c09c34c-kube-api-access-2z2z7\") pod \"test-curl-7684-140131340642352\" (UID: \"26589e33-9304-4fa3-b9cc-2a511c09c34c\") " pod="opendatahub/test-curl-7684-140131340642352"
Sep 09 20:46:42.229503 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:42.229452    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140131340642352"
Sep 09 20:46:42.502870 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:42.502849    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-140131340642352"]
Sep 09 20:46:42.505080 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:46:42.505050    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26589e33_9304_4fa3_b9cc_2a511c09c34c.slice/crio-7ea2fb5d36aef336db9336f5cb374744b1260c9c5364202e43685ccdcbda66ba WatchSource:0}: Error finding container 7ea2fb5d36aef336db9336f5cb374744b1260c9c5364202e43685ccdcbda66ba: Status 404 returned error can't find the container with id 7ea2fb5d36aef336db9336f5cb374744b1260c9c5364202e43685ccdcbda66ba
Sep 09 20:46:42.654567 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:42.654533    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-140131340642352" event={"ID":"26589e33-9304-4fa3-b9cc-2a511c09c34c","Type":"ContainerStarted","Data":"7ea2fb5d36aef336db9336f5cb374744b1260c9c5364202e43685ccdcbda66ba"}
Sep 09 20:46:44.244591 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:44.244565    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:46:44.671241 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:44.671206    2556 generic.go:356] "Generic (PLEG): container finished" podID="26589e33-9304-4fa3-b9cc-2a511c09c34c" containerID="103bb1fa63a015abf792c9f26a03b8ea05d1696b8deb72921850e9a9bf5a96c8" exitCode=0
Sep 09 20:46:44.671434 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:44.671289    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-140131340642352" event={"ID":"26589e33-9304-4fa3-b9cc-2a511c09c34c","Type":"ContainerDied","Data":"103bb1fa63a015abf792c9f26a03b8ea05d1696b8deb72921850e9a9bf5a96c8"}
Sep 09 20:46:45.805247 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:45.805225    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140131340642352"
Sep 09 20:46:45.850706 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:45.850680    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7684-140131340642352_26589e33-9304-4fa3-b9cc-2a511c09c34c/test-curl-7684-140131340642352/0.log"
Sep 09 20:46:45.866044 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:45.866013    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z2z7\" (UniqueName: \"kubernetes.io/projected/26589e33-9304-4fa3-b9cc-2a511c09c34c-kube-api-access-2z2z7\") pod \"26589e33-9304-4fa3-b9cc-2a511c09c34c\" (UID: \"26589e33-9304-4fa3-b9cc-2a511c09c34c\") "
Sep 09 20:46:45.868191 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:45.868166    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26589e33-9304-4fa3-b9cc-2a511c09c34c-kube-api-access-2z2z7" (OuterVolumeSpecName: "kube-api-access-2z2z7") pod "26589e33-9304-4fa3-b9cc-2a511c09c34c" (UID: "26589e33-9304-4fa3-b9cc-2a511c09c34c"). InnerVolumeSpecName "kube-api-access-2z2z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:46:45.877294 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:45.877268    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7684-140131340642352"]
Sep 09 20:46:45.883991 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:45.883973    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7684-140131340642352"]
Sep 09 20:46:45.967735 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:45.967677    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z2z7\" (UniqueName: \"kubernetes.io/projected/26589e33-9304-4fa3-b9cc-2a511c09c34c-kube-api-access-2z2z7\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:46:46.001619 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.001592    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7684-140131340644016"]
Sep 09 20:46:46.002000 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.001987    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26589e33-9304-4fa3-b9cc-2a511c09c34c" containerName="test-curl-7684-140131340642352"
Sep 09 20:46:46.002040 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.002002    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="26589e33-9304-4fa3-b9cc-2a511c09c34c" containerName="test-curl-7684-140131340642352"
Sep 09 20:46:46.002077 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.002066    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="26589e33-9304-4fa3-b9cc-2a511c09c34c" containerName="test-curl-7684-140131340642352"
Sep 09 20:46:46.005074 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.005055    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140131340644016"
Sep 09 20:46:46.020235 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.020214    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-140131340644016"]
Sep 09 20:46:46.068468 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.068436    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc7z9\" (UniqueName: \"kubernetes.io/projected/53c797bc-02aa-490d-bf03-238e101b8f07-kube-api-access-hc7z9\") pod \"test-curl-7684-140131340644016\" (UID: \"53c797bc-02aa-490d-bf03-238e101b8f07\") " pod="opendatahub/test-curl-7684-140131340644016"
Sep 09 20:46:46.169696 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.169671    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc7z9\" (UniqueName: \"kubernetes.io/projected/53c797bc-02aa-490d-bf03-238e101b8f07-kube-api-access-hc7z9\") pod \"test-curl-7684-140131340644016\" (UID: \"53c797bc-02aa-490d-bf03-238e101b8f07\") " pod="opendatahub/test-curl-7684-140131340644016"
Sep 09 20:46:46.178847 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.178825    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc7z9\" (UniqueName: \"kubernetes.io/projected/53c797bc-02aa-490d-bf03-238e101b8f07-kube-api-access-hc7z9\") pod \"test-curl-7684-140131340644016\" (UID: \"53c797bc-02aa-490d-bf03-238e101b8f07\") " pod="opendatahub/test-curl-7684-140131340644016"
Sep 09 20:46:46.313460 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.313434    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140131340644016"
Sep 09 20:46:46.522210 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.522179    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"]
Sep 09 20:46:46.526079 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.526056    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"
Sep 09 20:46:46.530449 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.530427    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-2b2343-serving-cert\""
Sep 09 20:46:46.530818 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.530801    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 20:46:46.531379 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.531362    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 20:46:46.532025 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.532010    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pzpjp\""
Sep 09 20:46:46.532251 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.532233    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 20:46:46.543473 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.543448    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"]
Sep 09 20:46:46.574220 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.574172    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f8c8e8a-d448-4a16-ab85-50518af36fcf-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq\" (UID: \"7f8c8e8a-d448-4a16-ab85-50518af36fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"
Sep 09 20:46:46.574220 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.574200    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnhft\" (UniqueName: \"kubernetes.io/projected/7f8c8e8a-d448-4a16-ab85-50518af36fcf-kube-api-access-pnhft\") pod \"maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq\" (UID: \"7f8c8e8a-d448-4a16-ab85-50518af36fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"
Sep 09 20:46:46.574220 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.574224    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f8c8e8a-d448-4a16-ab85-50518af36fcf-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq\" (UID: \"7f8c8e8a-d448-4a16-ab85-50518af36fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"
Sep 09 20:46:46.574391 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.574342    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f8c8e8a-d448-4a16-ab85-50518af36fcf-metrics-certs\") pod \"maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq\" (UID: \"7f8c8e8a-d448-4a16-ab85-50518af36fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"
Sep 09 20:46:46.589100 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.589081    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-140131340644016"]
Sep 09 20:46:46.590914 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:46:46.590884    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53c797bc_02aa_490d_bf03_238e101b8f07.slice/crio-2e02ce5220dca8c240e27055d3ff59faa2a869506b3967eedc20c81676052940 WatchSource:0}: Error finding container 2e02ce5220dca8c240e27055d3ff59faa2a869506b3967eedc20c81676052940: Status 404 returned error can't find the container with id 2e02ce5220dca8c240e27055d3ff59faa2a869506b3967eedc20c81676052940
Sep 09 20:46:46.675265 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.675225    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f8c8e8a-d448-4a16-ab85-50518af36fcf-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq\" (UID: \"7f8c8e8a-d448-4a16-ab85-50518af36fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"
Sep 09 20:46:46.675366 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.675276    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnhft\" (UniqueName: \"kubernetes.io/projected/7f8c8e8a-d448-4a16-ab85-50518af36fcf-kube-api-access-pnhft\") pod \"maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq\" (UID: \"7f8c8e8a-d448-4a16-ab85-50518af36fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"
Sep 09 20:46:46.675366 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.675310    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f8c8e8a-d448-4a16-ab85-50518af36fcf-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq\" (UID: \"7f8c8e8a-d448-4a16-ab85-50518af36fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"
Sep 09 20:46:46.675445 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.675367    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f8c8e8a-d448-4a16-ab85-50518af36fcf-metrics-certs\") pod \"maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq\" (UID: \"7f8c8e8a-d448-4a16-ab85-50518af36fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"
Sep 09 20:46:46.675803 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.675784    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f8c8e8a-d448-4a16-ab85-50518af36fcf-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq\" (UID: \"7f8c8e8a-d448-4a16-ab85-50518af36fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"
Sep 09 20:46:46.677793 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.677771    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f8c8e8a-d448-4a16-ab85-50518af36fcf-metrics-certs\") pod \"maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq\" (UID: \"7f8c8e8a-d448-4a16-ab85-50518af36fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"
Sep 09 20:46:46.677949 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.677931    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f8c8e8a-d448-4a16-ab85-50518af36fcf-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq\" (UID: \"7f8c8e8a-d448-4a16-ab85-50518af36fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"
Sep 09 20:46:46.682361 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.682344    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140131340642352"
Sep 09 20:46:46.682462 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.682373    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ea2fb5d36aef336db9336f5cb374744b1260c9c5364202e43685ccdcbda66ba"
Sep 09 20:46:46.683662 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.683638    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-140131340644016" event={"ID":"53c797bc-02aa-490d-bf03-238e101b8f07","Type":"ContainerStarted","Data":"2e02ce5220dca8c240e27055d3ff59faa2a869506b3967eedc20c81676052940"}
Sep 09 20:46:46.687128 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.687109    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnhft\" (UniqueName: \"kubernetes.io/projected/7f8c8e8a-d448-4a16-ab85-50518af36fcf-kube-api-access-pnhft\") pod \"maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq\" (UID: \"7f8c8e8a-d448-4a16-ab85-50518af36fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"
Sep 09 20:46:46.695320 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:46:46.695298    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7684-140131340642352\" is forbidden: User \"system:node:ip-10-0-135-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-117.ec2.internal' and this object" podUID="26589e33-9304-4fa3-b9cc-2a511c09c34c" pod="opendatahub/test-curl-7684-140131340642352"
Sep 09 20:46:46.838232 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.838172    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"
Sep 09 20:46:46.896523 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:46.896491    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26589e33-9304-4fa3-b9cc-2a511c09c34c" path="/var/lib/kubelet/pods/26589e33-9304-4fa3-b9cc-2a511c09c34c/volumes"
Sep 09 20:46:47.125653 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:47.125632    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"]
Sep 09 20:46:47.127360 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:46:47.127305    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8c8e8a_d448_4a16_ab85_50518af36fcf.slice/crio-ebb75e8ee7c0705bc39ad9af7ecb00cf229d70187b7070edcef228cdad2bd1ff WatchSource:0}: Error finding container ebb75e8ee7c0705bc39ad9af7ecb00cf229d70187b7070edcef228cdad2bd1ff: Status 404 returned error can't find the container with id ebb75e8ee7c0705bc39ad9af7ecb00cf229d70187b7070edcef228cdad2bd1ff
Sep 09 20:46:47.689735 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:47.689681    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq" event={"ID":"7f8c8e8a-d448-4a16-ab85-50518af36fcf","Type":"ContainerStarted","Data":"ebb75e8ee7c0705bc39ad9af7ecb00cf229d70187b7070edcef228cdad2bd1ff"}
Sep 09 20:46:47.691269 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:47.691242    2556 generic.go:356] "Generic (PLEG): container finished" podID="53c797bc-02aa-490d-bf03-238e101b8f07" containerID="63b522ba45ceff81381fbe72604b5216db5268366bf36bae5113cf817e3a534a" exitCode=0
Sep 09 20:46:47.691392 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:47.691330    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-140131340644016" event={"ID":"53c797bc-02aa-490d-bf03-238e101b8f07","Type":"ContainerDied","Data":"63b522ba45ceff81381fbe72604b5216db5268366bf36bae5113cf817e3a534a"}
Sep 09 20:46:47.773746 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:47.773712    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9"]
Sep 09 20:46:47.777465 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:47.777442    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9"
Sep 09 20:46:47.780162 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:47.780134    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-2b2343\""
Sep 09 20:46:47.780262 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:47.780210    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-2b2343-dockercfg-bnz85\""
Sep 09 20:46:47.787135 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:47.787108    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c1d67e-0033-4ad7-8c03-58e7fc069e0a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9\" (UID: \"f3c1d67e-0033-4ad7-8c03-58e7fc069e0a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9"
Sep 09 20:46:47.787341 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:47.787310    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8xzx\" (UniqueName: \"kubernetes.io/projected/f3c1d67e-0033-4ad7-8c03-58e7fc069e0a-kube-api-access-m8xzx\") pod \"payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9\" (UID: \"f3c1d67e-0033-4ad7-8c03-58e7fc069e0a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9"
Sep 09 20:46:47.788188 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:47.788169    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9"]
Sep 09 20:46:47.887895 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:47.887855    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c1d67e-0033-4ad7-8c03-58e7fc069e0a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9\" (UID: \"f3c1d67e-0033-4ad7-8c03-58e7fc069e0a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9"
Sep 09 20:46:47.888355 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:47.887932    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8xzx\" (UniqueName: \"kubernetes.io/projected/f3c1d67e-0033-4ad7-8c03-58e7fc069e0a-kube-api-access-m8xzx\") pod \"payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9\" (UID: \"f3c1d67e-0033-4ad7-8c03-58e7fc069e0a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9"
Sep 09 20:46:47.888671 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:47.888647    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c1d67e-0033-4ad7-8c03-58e7fc069e0a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9\" (UID: \"f3c1d67e-0033-4ad7-8c03-58e7fc069e0a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9"
Sep 09 20:46:47.896406 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:47.896380    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8xzx\" (UniqueName: \"kubernetes.io/projected/f3c1d67e-0033-4ad7-8c03-58e7fc069e0a-kube-api-access-m8xzx\") pod \"payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9\" (UID: \"f3c1d67e-0033-4ad7-8c03-58e7fc069e0a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9"
Sep 09 20:46:48.099332 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:48.099299    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9"
Sep 09 20:46:48.380656 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:48.380582    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9"]
Sep 09 20:46:48.382827 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:46:48.382794    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c1d67e_0033_4ad7_8c03_58e7fc069e0a.slice/crio-3a32a00a044e6d6e1b12770a53a2915707075439dc1ae192f3a8bc6c8d1c226b WatchSource:0}: Error finding container 3a32a00a044e6d6e1b12770a53a2915707075439dc1ae192f3a8bc6c8d1c226b: Status 404 returned error can't find the container with id 3a32a00a044e6d6e1b12770a53a2915707075439dc1ae192f3a8bc6c8d1c226b
Sep 09 20:46:48.698978 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:48.698863    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9" event={"ID":"f3c1d67e-0033-4ad7-8c03-58e7fc069e0a","Type":"ContainerStarted","Data":"3a32a00a044e6d6e1b12770a53a2915707075439dc1ae192f3a8bc6c8d1c226b"}
Sep 09 20:46:48.939681 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:48.939658    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140131340644016"
Sep 09 20:46:48.968107 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:48.968089    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7684-140131340644016_53c797bc-02aa-490d-bf03-238e101b8f07/test-curl-7684-140131340644016/0.log"
Sep 09 20:46:48.997690 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:48.997663    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7684-140131340644016"]
Sep 09 20:46:48.999099 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:48.999071    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc7z9\" (UniqueName: \"kubernetes.io/projected/53c797bc-02aa-490d-bf03-238e101b8f07-kube-api-access-hc7z9\") pod \"53c797bc-02aa-490d-bf03-238e101b8f07\" (UID: \"53c797bc-02aa-490d-bf03-238e101b8f07\") "
Sep 09 20:46:49.001159 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.001136    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7684-140131340644016"]
Sep 09 20:46:49.001320 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.001296    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c797bc-02aa-490d-bf03-238e101b8f07-kube-api-access-hc7z9" (OuterVolumeSpecName: "kube-api-access-hc7z9") pod "53c797bc-02aa-490d-bf03-238e101b8f07" (UID: "53c797bc-02aa-490d-bf03-238e101b8f07"). InnerVolumeSpecName "kube-api-access-hc7z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:46:49.100470 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.100433    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc7z9\" (UniqueName: \"kubernetes.io/projected/53c797bc-02aa-490d-bf03-238e101b8f07-kube-api-access-hc7z9\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:46:49.706577 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.706539    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq" event={"ID":"7f8c8e8a-d448-4a16-ab85-50518af36fcf","Type":"ContainerStarted","Data":"345ef4467df4892b97d4ba3a67753319e51da8a695a73a6c62baace5cac9915a"}
Sep 09 20:46:49.706778 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.706679    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"
Sep 09 20:46:49.707992 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.707970    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e02ce5220dca8c240e27055d3ff59faa2a869506b3967eedc20c81676052940"
Sep 09 20:46:49.708124 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.707996    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140131340644016"
Sep 09 20:46:49.733230 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.733139    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq" podStartSLOduration=1.895057126 podStartE2EDuration="3.733123895s" totalImagesPullingTime="1.838066769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:46:49.731032402 +0000 UTC m=+855.410992710" watchObservedRunningTime="2026-09-09 20:46:49.733123895 +0000 UTC m=+855.413084204"
Sep 09 20:46:49.733340 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:46:49.733306    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7684-140131340644016\" is forbidden: User \"system:node:ip-10-0-135-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-117.ec2.internal' and this object" podUID="53c797bc-02aa-490d-bf03-238e101b8f07" pod="opendatahub/test-curl-7684-140131340644016"
Sep 09 20:46:49.871020 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.870978    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"]
Sep 09 20:46:49.871581 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.871557    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53c797bc-02aa-490d-bf03-238e101b8f07" containerName="test-curl-7684-140131340644016"
Sep 09 20:46:49.871581 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.871576    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="53c797bc-02aa-490d-bf03-238e101b8f07" containerName="test-curl-7684-140131340644016"
Sep 09 20:46:49.871738 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.871699    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="53c797bc-02aa-490d-bf03-238e101b8f07" containerName="test-curl-7684-140131340644016"
Sep 09 20:46:49.875267 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.875232    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:49.878307 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.878260    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 20:46:49.878616 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.878575    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-2beba0-openshift-default-dockercfg-2ddg9\""
Sep 09 20:46:49.893978 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.893954    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"]
Sep 09 20:46:49.907119 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.907095    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-data\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:49.907248 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.907138    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-credential-socket\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:49.907248 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.907170    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb9f926-a0c6-40df-b3af-7bba52226164-istiod-ca-cert\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:49.907370 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.907251    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-podinfo\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:49.907370 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.907296    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-workload-socket\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:49.907370 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.907333    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-token\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:49.907535 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.907369    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-envoy\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:49.907535 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.907397    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-workload-certs\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:49.907535 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:49.907459    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkcv4\" (UniqueName: \"kubernetes.io/projected/4eb9f926-a0c6-40df-b3af-7bba52226164-kube-api-access-rkcv4\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:49.911279 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:46:49.911238    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7684-140131340644016\" is forbidden: User \"system:node:ip-10-0-135-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-117.ec2.internal' and this object" podUID="53c797bc-02aa-490d-bf03-238e101b8f07" pod="opendatahub/test-curl-7684-140131340644016"
Sep 09 20:46:50.008625 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.008589    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-podinfo\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:50.009111 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.008727    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-workload-socket\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:50.009111 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.008771    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-token\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:50.009111 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.008820    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-envoy\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:50.009111 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.008850    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-workload-certs\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:50.009111 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.008894    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkcv4\" (UniqueName: \"kubernetes.io/projected/4eb9f926-a0c6-40df-b3af-7bba52226164-kube-api-access-rkcv4\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:50.009111 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.009044    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-data\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:50.009111 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.009079    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-credential-socket\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:50.009111 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.009112    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb9f926-a0c6-40df-b3af-7bba52226164-istiod-ca-cert\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:50.009557 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.009178    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-workload-socket\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:50.009557 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.009358    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-data\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:50.009663 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.009598    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-credential-socket\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:50.009663 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.009636    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-workload-certs\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:50.009855 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.009804    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb9f926-a0c6-40df-b3af-7bba52226164-istiod-ca-cert\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:50.011806 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.011771    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-envoy\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:50.012142 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.012108    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-podinfo\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:50.017753 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.017729    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-token\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:50.018718 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.018697    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkcv4\" (UniqueName: \"kubernetes.io/projected/4eb9f926-a0c6-40df-b3af-7bba52226164-kube-api-access-rkcv4\") pod \"e2e-autores-2beba0-openshift-default-696968fc9d-d68zb\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") " pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:50.187494 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.187448    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:50.469728 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.469642    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"]
Sep 09 20:46:50.472836 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:46:50.472808    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb9f926_a0c6_40df_b3af_7bba52226164.slice/crio-beeb0ccc0ea1227f5cd5abcb722421b4e01a3693b03464bf687b1bc14235cb8b WatchSource:0}: Error finding container beeb0ccc0ea1227f5cd5abcb722421b4e01a3693b03464bf687b1bc14235cb8b: Status 404 returned error can't find the container with id beeb0ccc0ea1227f5cd5abcb722421b4e01a3693b03464bf687b1bc14235cb8b
Sep 09 20:46:50.715834 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.715798    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9" event={"ID":"f3c1d67e-0033-4ad7-8c03-58e7fc069e0a","Type":"ContainerStarted","Data":"63171f67cce892ae29c61d87a8930683df1d5e1db2f2a8e01433e8846d423ca0"}
Sep 09 20:46:50.715989 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.715866    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9"
Sep 09 20:46:50.717029 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.717005    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb" event={"ID":"4eb9f926-a0c6-40df-b3af-7bba52226164","Type":"ContainerStarted","Data":"beeb0ccc0ea1227f5cd5abcb722421b4e01a3693b03464bf687b1bc14235cb8b"}
Sep 09 20:46:50.764501 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.764456    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9" podStartSLOduration=1.9839320379999998 podStartE2EDuration="3.764434489s" totalImagesPullingTime="1.780502451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:46:50.764280292 +0000 UTC m=+856.444240601" watchObservedRunningTime="2026-09-09 20:46:50.764434489 +0000 UTC m=+856.444394797"
Sep 09 20:46:50.767973 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:46:50.767951    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7684-140131340644016\" is forbidden: User \"system:node:ip-10-0-135-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-117.ec2.internal' and this object" podUID="53c797bc-02aa-490d-bf03-238e101b8f07" pod="opendatahub/test-curl-7684-140131340644016"
Sep 09 20:46:50.900103 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:50.900072    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53c797bc-02aa-490d-bf03-238e101b8f07" path="/var/lib/kubelet/pods/53c797bc-02aa-490d-bf03-238e101b8f07/volumes"
Sep 09 20:46:53.175850 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:53.175809    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:46:53.176253 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:53.175881    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:46:53.176253 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:53.175960    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:46:53.736944 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:53.736899    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb" event={"ID":"4eb9f926-a0c6-40df-b3af-7bba52226164","Type":"ContainerStarted","Data":"cdb6d3122b8de657a4e9efb1ec76e8593645f29c7e96b6ad22b6ae32a43efa4e"}
Sep 09 20:46:53.774703 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:53.774648    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb" podStartSLOduration=2.074468339 podStartE2EDuration="4.774633678s" totalImagesPullingTime="2.700165339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:46:53.772280934 +0000 UTC m=+859.452241268" watchObservedRunningTime="2026-09-09 20:46:53.774633678 +0000 UTC m=+859.454593986"
Sep 09 20:46:54.188547 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:54.188464    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:54.193321 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:54.193282    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:54.743299 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:54.743265    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:54.744305 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:54.744280    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:46:55.432095 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:55.432062    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs"]
Sep 09 20:46:55.435752 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:55.435738    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs"
Sep 09 20:46:55.438503 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:55.438468    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-04f122cb\""
Sep 09 20:46:55.438681 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:55.438664    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-04f122cb-dockercfg-82v6f\""
Sep 09 20:46:55.446358 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:55.446337    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs"]
Sep 09 20:46:55.558032 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:55.558005    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d3822b-e912-4127-9e4c-b3a6c18d296f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs\" (UID: \"78d3822b-e912-4127-9e4c-b3a6c18d296f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs"
Sep 09 20:46:55.558177 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:55.558057    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xlrt\" (UniqueName: \"kubernetes.io/projected/78d3822b-e912-4127-9e4c-b3a6c18d296f-kube-api-access-5xlrt\") pod \"payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs\" (UID: \"78d3822b-e912-4127-9e4c-b3a6c18d296f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs"
Sep 09 20:46:55.658772 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:55.658741    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d3822b-e912-4127-9e4c-b3a6c18d296f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs\" (UID: \"78d3822b-e912-4127-9e4c-b3a6c18d296f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs"
Sep 09 20:46:55.658925 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:55.658784    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xlrt\" (UniqueName: \"kubernetes.io/projected/78d3822b-e912-4127-9e4c-b3a6c18d296f-kube-api-access-5xlrt\") pod \"payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs\" (UID: \"78d3822b-e912-4127-9e4c-b3a6c18d296f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs"
Sep 09 20:46:55.659488 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:55.659467    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d3822b-e912-4127-9e4c-b3a6c18d296f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs\" (UID: \"78d3822b-e912-4127-9e4c-b3a6c18d296f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs"
Sep 09 20:46:55.667854 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:55.667827    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xlrt\" (UniqueName: \"kubernetes.io/projected/78d3822b-e912-4127-9e4c-b3a6c18d296f-kube-api-access-5xlrt\") pod \"payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs\" (UID: \"78d3822b-e912-4127-9e4c-b3a6c18d296f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs"
Sep 09 20:46:55.722543 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:55.722493    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"
Sep 09 20:46:55.749560 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:55.749530    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs"
Sep 09 20:46:56.032675 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:56.032650    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs"]
Sep 09 20:46:56.034849 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:46:56.034817    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d3822b_e912_4127_9e4c_b3a6c18d296f.slice/crio-e17169cd29cfe867da45df4c107cff5a50a329e76c7524bdab933af9dd78b27f WatchSource:0}: Error finding container e17169cd29cfe867da45df4c107cff5a50a329e76c7524bdab933af9dd78b27f: Status 404 returned error can't find the container with id e17169cd29cfe867da45df4c107cff5a50a329e76c7524bdab933af9dd78b27f
Sep 09 20:46:56.762360 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:56.762330    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs" event={"ID":"78d3822b-e912-4127-9e4c-b3a6c18d296f","Type":"ContainerStarted","Data":"fcfaa517071f27ba863088d01c10647fed42cd57d54463508a1ad805a44a4ea2"}
Sep 09 20:46:56.762713 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:56.762378    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs" event={"ID":"78d3822b-e912-4127-9e4c-b3a6c18d296f","Type":"ContainerStarted","Data":"e17169cd29cfe867da45df4c107cff5a50a329e76c7524bdab933af9dd78b27f"}
Sep 09 20:46:56.762713 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:56.762419    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs"
Sep 09 20:46:56.786014 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:56.785969    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs" podStartSLOduration=1.785956803 podStartE2EDuration="1.785956803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:46:56.783796273 +0000 UTC m=+862.463756581" watchObservedRunningTime="2026-09-09 20:46:56.785956803 +0000 UTC m=+862.465917112"
Sep 09 20:46:58.732887 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:58.732853    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5"]
Sep 09 20:46:58.736742 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:58.736725    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5"
Sep 09 20:46:58.739744 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:58.739725    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-2beba0\""
Sep 09 20:46:58.739805 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:58.739761    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-2beba0-dockercfg-bd44j\""
Sep 09 20:46:58.749043 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:58.749016    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5"]
Sep 09 20:46:58.786818 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:58.786790    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67skl\" (UniqueName: \"kubernetes.io/projected/2caa6df3-5b63-4212-b03c-8909bb2f4f92-kube-api-access-67skl\") pod \"payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5\" (UID: \"2caa6df3-5b63-4212-b03c-8909bb2f4f92\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5"
Sep 09 20:46:58.786954 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:58.786846    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2caa6df3-5b63-4212-b03c-8909bb2f4f92-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5\" (UID: \"2caa6df3-5b63-4212-b03c-8909bb2f4f92\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5"
Sep 09 20:46:58.888152 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:58.888119    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67skl\" (UniqueName: \"kubernetes.io/projected/2caa6df3-5b63-4212-b03c-8909bb2f4f92-kube-api-access-67skl\") pod \"payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5\" (UID: \"2caa6df3-5b63-4212-b03c-8909bb2f4f92\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5"
Sep 09 20:46:58.888293 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:58.888166    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2caa6df3-5b63-4212-b03c-8909bb2f4f92-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5\" (UID: \"2caa6df3-5b63-4212-b03c-8909bb2f4f92\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5"
Sep 09 20:46:58.888830 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:58.888804    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2caa6df3-5b63-4212-b03c-8909bb2f4f92-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5\" (UID: \"2caa6df3-5b63-4212-b03c-8909bb2f4f92\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5"
Sep 09 20:46:58.896531 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:58.896504    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67skl\" (UniqueName: \"kubernetes.io/projected/2caa6df3-5b63-4212-b03c-8909bb2f4f92-kube-api-access-67skl\") pod \"payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5\" (UID: \"2caa6df3-5b63-4212-b03c-8909bb2f4f92\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5"
Sep 09 20:46:59.047743 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:59.047682    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5"
Sep 09 20:46:59.323191 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:59.323168    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5"]
Sep 09 20:46:59.325139 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:46:59.325108    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2caa6df3_5b63_4212_b03c_8909bb2f4f92.slice/crio-b41cb25330920a375d92885e0a92100fbd307e95e874200b31c51cf6a7d496bb WatchSource:0}: Error finding container b41cb25330920a375d92885e0a92100fbd307e95e874200b31c51cf6a7d496bb: Status 404 returned error can't find the container with id b41cb25330920a375d92885e0a92100fbd307e95e874200b31c51cf6a7d496bb
Sep 09 20:46:59.789042 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:59.789002    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5" event={"ID":"2caa6df3-5b63-4212-b03c-8909bb2f4f92","Type":"ContainerStarted","Data":"3b43e54a0f1dbc43d34bdfb6e1b2fbf2a483ca715b088a0e83ffaacc9ffb7822"}
Sep 09 20:46:59.789498 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:59.789099    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5" event={"ID":"2caa6df3-5b63-4212-b03c-8909bb2f4f92","Type":"ContainerStarted","Data":"b41cb25330920a375d92885e0a92100fbd307e95e874200b31c51cf6a7d496bb"}
Sep 09 20:46:59.789498 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:59.789300    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5"
Sep 09 20:46:59.810036 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:46:59.809992    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5" podStartSLOduration=1.8099778149999999 podStartE2EDuration="1.809977815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:46:59.807348417 +0000 UTC m=+865.487308728" watchObservedRunningTime="2026-09-09 20:46:59.809977815 +0000 UTC m=+865.489938169"
Sep 09 20:47:01.726360 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:01.726332    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9"
Sep 09 20:47:04.909067 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:04.909032    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:47:07.772122 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:07.772084    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs"
Sep 09 20:47:10.796544 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:10.796506    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5"
Sep 09 20:47:15.872955 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:15.872921    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"]
Sep 09 20:47:15.875599 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:15.875577    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:15.876552 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:15.876528    2556 generic.go:356] "Generic (PLEG): container finished" podID="4eb9f926-a0c6-40df-b3af-7bba52226164" containerID="cdb6d3122b8de657a4e9efb1ec76e8593645f29c7e96b6ad22b6ae32a43efa4e" exitCode=0
Sep 09 20:47:15.876634 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:15.876601    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb" event={"ID":"4eb9f926-a0c6-40df-b3af-7bba52226164","Type":"ContainerDied","Data":"cdb6d3122b8de657a4e9efb1ec76e8593645f29c7e96b6ad22b6ae32a43efa4e"}
Sep 09 20:47:15.877040 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:15.877022    2556 scope.go:117] "RemoveContainer" containerID="cdb6d3122b8de657a4e9efb1ec76e8593645f29c7e96b6ad22b6ae32a43efa4e"
Sep 09 20:47:15.878111 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:15.878084    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:47:15.878190 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:15.878143    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:47:15.878190 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:15.878174    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:47:15.880305 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:15.880288    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b2dd30-openshift-default-dockercfg-7mnxm\""
Sep 09 20:47:15.911354 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:15.908006    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"]
Sep 09 20:47:15.958759 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:15.958733    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-envoy\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:15.959035 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:15.959008    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-workload-certs\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:15.959163 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:15.959146    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56sck\" (UniqueName: \"kubernetes.io/projected/fbbd858d-84a8-40c5-a9ea-036ee3597316-kube-api-access-56sck\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:15.959229 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:15.959202    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-token\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:15.959281 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:15.959227    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbbd858d-84a8-40c5-a9ea-036ee3597316-istiod-ca-cert\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:15.959281 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:15.959254    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-credential-socket\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:15.959386 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:15.959284    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-podinfo\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:15.959386 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:15.959315    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-data\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:15.959386 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:15.959352    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-workload-socket\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:16.060630 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.060595    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-workload-certs\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:16.060816 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.060719    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56sck\" (UniqueName: \"kubernetes.io/projected/fbbd858d-84a8-40c5-a9ea-036ee3597316-kube-api-access-56sck\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:16.060816 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.060767    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-token\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:16.060816 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.060794    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbbd858d-84a8-40c5-a9ea-036ee3597316-istiod-ca-cert\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:16.061007 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.060826    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-credential-socket\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:16.061007 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.060853    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-podinfo\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:16.061007 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.060882    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-data\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:16.061171 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.061052    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-workload-socket\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:16.061171 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.061097    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-envoy\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:16.061280 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.061213    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-workload-certs\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:16.061280 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.061267    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-credential-socket\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:16.061439 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.061413    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-data\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:16.061562 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.061485    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-workload-socket\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:16.062057 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.062037    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbbd858d-84a8-40c5-a9ea-036ee3597316-istiod-ca-cert\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:16.063840 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.063810    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-envoy\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:16.063964 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.063868    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-podinfo\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:16.077602 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.077580    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56sck\" (UniqueName: \"kubernetes.io/projected/fbbd858d-84a8-40c5-a9ea-036ee3597316-kube-api-access-56sck\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:16.077678 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.077608    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-token\") pod \"e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") " pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:16.187085 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.187011    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:16.466763 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.466544    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"]
Sep 09 20:47:16.469350 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:47:16.469327    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbbd858d_84a8_40c5_a9ea_036ee3597316.slice/crio-d5a3d8ba640391fa1d0f4c39389a27127c8435b67830d2f20d85377b0f16a0fb WatchSource:0}: Error finding container d5a3d8ba640391fa1d0f4c39389a27127c8435b67830d2f20d85377b0f16a0fb: Status 404 returned error can't find the container with id d5a3d8ba640391fa1d0f4c39389a27127c8435b67830d2f20d85377b0f16a0fb
Sep 09 20:47:16.471875 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.471843    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:47:16.472006 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.471938    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:47:16.472006 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.471985    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:47:16.888076 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.888031    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz" event={"ID":"fbbd858d-84a8-40c5-a9ea-036ee3597316","Type":"ContainerStarted","Data":"8083f01b6fa208c0eb6e67037b8ad02e2cfcc6d0cd3daafc03f01df602776080"}
Sep 09 20:47:16.888589 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.888104    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz" event={"ID":"fbbd858d-84a8-40c5-a9ea-036ee3597316","Type":"ContainerStarted","Data":"d5a3d8ba640391fa1d0f4c39389a27127c8435b67830d2f20d85377b0f16a0fb"}
Sep 09 20:47:16.891814 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.891781    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb" event={"ID":"4eb9f926-a0c6-40df-b3af-7bba52226164","Type":"ContainerStarted","Data":"e9f4b1939bc7bdeaf7a8842b217cba35377b93fa0be9eeea6676b3b120eb7a24"}
Sep 09 20:47:16.917275 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:16.917218    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz" podStartSLOduration=1.917202329 podStartE2EDuration="1.917202329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:47:16.914201987 +0000 UTC m=+882.594162299" watchObservedRunningTime="2026-09-09 20:47:16.917202329 +0000 UTC m=+882.597162637"
Sep 09 20:47:17.189608 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:17.189502    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:17.189608 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:17.189583    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:47:17.189949 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:17.189921    2556 patch_prober.go:28] interesting pod/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 09 20:47:17.190044 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:17.190002    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb" podUID="4eb9f926-a0c6-40df-b3af-7bba52226164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 09 20:47:17.195105 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:17.195076    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:17.898260 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:17.898214    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:17.899129 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:17.899110    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:18.011179 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:18.011144    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f122cb-jxc2s"]
Sep 09 20:47:18.013650 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:18.013628    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f122cb-jxc2s"
Sep 09 20:47:18.016710 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:18.016685    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsks9\""
Sep 09 20:47:18.023486 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:18.023453    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f122cb-jxc2s"]
Sep 09 20:47:18.188866 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:18.188766    2556 patch_prober.go:28] interesting pod/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 09 20:47:18.188866 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:18.188833    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb" podUID="4eb9f926-a0c6-40df-b3af-7bba52226164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 09 20:47:18.197457 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:18.197425    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dbeed26-58e2-4d5c-a493-cbadebe3c484-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-04f122cb-jxc2s\" (UID: \"4dbeed26-58e2-4d5c-a493-cbadebe3c484\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f122cb-jxc2s"
Sep 09 20:47:18.299167 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:18.299136    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dbeed26-58e2-4d5c-a493-cbadebe3c484-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-04f122cb-jxc2s\" (UID: \"4dbeed26-58e2-4d5c-a493-cbadebe3c484\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f122cb-jxc2s"
Sep 09 20:47:18.299809 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:18.299787    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dbeed26-58e2-4d5c-a493-cbadebe3c484-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-04f122cb-jxc2s\" (UID: \"4dbeed26-58e2-4d5c-a493-cbadebe3c484\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f122cb-jxc2s"
Sep 09 20:47:18.325434 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:18.325411    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f122cb-jxc2s"
Sep 09 20:47:18.627802 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:18.627770    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f122cb-jxc2s"]
Sep 09 20:47:18.631397 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:47:18.631353    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dbeed26_58e2_4d5c_a493_cbadebe3c484.slice/crio-255405d8e52deb0542dce54b4c271cc239a4c072fc23924def87ba1704e2108b WatchSource:0}: Error finding container 255405d8e52deb0542dce54b4c271cc239a4c072fc23924def87ba1704e2108b: Status 404 returned error can't find the container with id 255405d8e52deb0542dce54b4c271cc239a4c072fc23924def87ba1704e2108b
Sep 09 20:47:18.908291 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:18.908257    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f122cb-jxc2s" event={"ID":"4dbeed26-58e2-4d5c-a493-cbadebe3c484","Type":"ContainerStarted","Data":"255405d8e52deb0542dce54b4c271cc239a4c072fc23924def87ba1704e2108b"}
Sep 09 20:47:19.187994 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:19.187955    2556 patch_prober.go:28] interesting pod/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 09 20:47:19.188270 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:19.188241    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb" podUID="4eb9f926-a0c6-40df-b3af-7bba52226164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 09 20:47:19.915777 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:19.915694    2556 generic.go:356] "Generic (PLEG): container finished" podID="4dbeed26-58e2-4d5c-a493-cbadebe3c484" containerID="557dfcf2cc32bb2d5fd3199ecf969f0692bee36e0f84e7aa99003771c2be1338" exitCode=0
Sep 09 20:47:19.916184 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:19.915787    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f122cb-jxc2s" event={"ID":"4dbeed26-58e2-4d5c-a493-cbadebe3c484","Type":"ContainerDied","Data":"557dfcf2cc32bb2d5fd3199ecf969f0692bee36e0f84e7aa99003771c2be1338"}
Sep 09 20:47:20.188211 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:20.188117    2556 patch_prober.go:28] interesting pod/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 09 20:47:20.188478 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:20.188436    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb" podUID="4eb9f926-a0c6-40df-b3af-7bba52226164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 09 20:47:20.617397 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:20.617373    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"]
Sep 09 20:47:20.620230 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:20.620213    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"
Sep 09 20:47:20.624038 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:20.624015    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b2dd30-serving-cert\""
Sep 09 20:47:20.639733 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:20.639710    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"]
Sep 09 20:47:20.721011 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:20.720984    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsbpf\" (UniqueName: \"kubernetes.io/projected/59076416-48b5-44bf-93cd-15e75a5b6e60-kube-api-access-tsbpf\") pod \"maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn\" (UID: \"59076416-48b5-44bf-93cd-15e75a5b6e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"
Sep 09 20:47:20.721245 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:20.721225    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59076416-48b5-44bf-93cd-15e75a5b6e60-maas-api-tls\") pod \"maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn\" (UID: \"59076416-48b5-44bf-93cd-15e75a5b6e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"
Sep 09 20:47:20.721430 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:20.721414    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59076416-48b5-44bf-93cd-15e75a5b6e60-metrics-certs\") pod \"maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn\" (UID: \"59076416-48b5-44bf-93cd-15e75a5b6e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"
Sep 09 20:47:20.721526 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:20.721496    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59076416-48b5-44bf-93cd-15e75a5b6e60-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn\" (UID: \"59076416-48b5-44bf-93cd-15e75a5b6e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"
Sep 09 20:47:20.822595 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:20.822568    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsbpf\" (UniqueName: \"kubernetes.io/projected/59076416-48b5-44bf-93cd-15e75a5b6e60-kube-api-access-tsbpf\") pod \"maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn\" (UID: \"59076416-48b5-44bf-93cd-15e75a5b6e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"
Sep 09 20:47:20.822828 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:20.822808    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59076416-48b5-44bf-93cd-15e75a5b6e60-maas-api-tls\") pod \"maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn\" (UID: \"59076416-48b5-44bf-93cd-15e75a5b6e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"
Sep 09 20:47:20.823018 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:20.823001    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59076416-48b5-44bf-93cd-15e75a5b6e60-metrics-certs\") pod \"maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn\" (UID: \"59076416-48b5-44bf-93cd-15e75a5b6e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"
Sep 09 20:47:20.823182 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:20.823165    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59076416-48b5-44bf-93cd-15e75a5b6e60-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn\" (UID: \"59076416-48b5-44bf-93cd-15e75a5b6e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"
Sep 09 20:47:20.823733 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:20.823712    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59076416-48b5-44bf-93cd-15e75a5b6e60-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn\" (UID: \"59076416-48b5-44bf-93cd-15e75a5b6e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"
Sep 09 20:47:20.825343 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:20.825323    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59076416-48b5-44bf-93cd-15e75a5b6e60-maas-api-tls\") pod \"maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn\" (UID: \"59076416-48b5-44bf-93cd-15e75a5b6e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"
Sep 09 20:47:20.825597 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:20.825578    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59076416-48b5-44bf-93cd-15e75a5b6e60-metrics-certs\") pod \"maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn\" (UID: \"59076416-48b5-44bf-93cd-15e75a5b6e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"
Sep 09 20:47:20.832068 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:20.832046    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsbpf\" (UniqueName: \"kubernetes.io/projected/59076416-48b5-44bf-93cd-15e75a5b6e60-kube-api-access-tsbpf\") pod \"maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn\" (UID: \"59076416-48b5-44bf-93cd-15e75a5b6e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"
Sep 09 20:47:20.932272 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:20.932191    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"
Sep 09 20:47:21.099964 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.099935    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f122cb-jxc2s"
Sep 09 20:47:21.126144 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.126111    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dbeed26-58e2-4d5c-a493-cbadebe3c484-maas-api-service-ca\") pod \"4dbeed26-58e2-4d5c-a493-cbadebe3c484\" (UID: \"4dbeed26-58e2-4d5c-a493-cbadebe3c484\") "
Sep 09 20:47:21.126466 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.126444    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dbeed26-58e2-4d5c-a493-cbadebe3c484-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4dbeed26-58e2-4d5c-a493-cbadebe3c484" (UID: "4dbeed26-58e2-4d5c-a493-cbadebe3c484"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:47:21.126632 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.126617    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dbeed26-58e2-4d5c-a493-cbadebe3c484-maas-api-service-ca\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:47:21.193062 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.192986    2556 patch_prober.go:28] interesting pod/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 09 20:47:21.193062 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.193035    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb" podUID="4eb9f926-a0c6-40df-b3af-7bba52226164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 09 20:47:21.251011 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.250989    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"]
Sep 09 20:47:21.253128 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:47:21.253101    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59076416_48b5_44bf_93cd_15e75a5b6e60.slice/crio-999798e97bef21d47d14cf61be37c3c3f5ad2f765c8db6b5c0e71356976bf4f1 WatchSource:0}: Error finding container 999798e97bef21d47d14cf61be37c3c3f5ad2f765c8db6b5c0e71356976bf4f1: Status 404 returned error can't find the container with id 999798e97bef21d47d14cf61be37c3c3f5ad2f765c8db6b5c0e71356976bf4f1
Sep 09 20:47:21.418332 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.418300    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l"]
Sep 09 20:47:21.418741 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.418729    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dbeed26-58e2-4d5c-a493-cbadebe3c484" containerName="revoke-keys"
Sep 09 20:47:21.418741 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.418742    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dbeed26-58e2-4d5c-a493-cbadebe3c484" containerName="revoke-keys"
Sep 09 20:47:21.418846 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.418813    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dbeed26-58e2-4d5c-a493-cbadebe3c484" containerName="revoke-keys"
Sep 09 20:47:21.421211 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.421195    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l"
Sep 09 20:47:21.425932 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.425845    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b2dd30\""
Sep 09 20:47:21.429754 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.429729    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b0cd0a-cf56-4821-915a-117f5b75556e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l\" (UID: \"63b0cd0a-cf56-4821-915a-117f5b75556e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l"
Sep 09 20:47:21.429866 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.429761    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htlrk\" (UniqueName: \"kubernetes.io/projected/63b0cd0a-cf56-4821-915a-117f5b75556e-kube-api-access-htlrk\") pod \"payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l\" (UID: \"63b0cd0a-cf56-4821-915a-117f5b75556e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l"
Sep 09 20:47:21.439139 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.439116    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l"]
Sep 09 20:47:21.501423 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.501392    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx"]
Sep 09 20:47:21.504056 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.504040    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx"
Sep 09 20:47:21.507492 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.507472    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b2dd30-dockercfg-sklpv\""
Sep 09 20:47:21.516854 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.516542    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx"]
Sep 09 20:47:21.531289 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.531263    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhbjc\" (UniqueName: \"kubernetes.io/projected/997cb48e-b246-4ae0-b2d2-86fe8fe7ef00-kube-api-access-jhbjc\") pod \"payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx\" (UID: \"997cb48e-b246-4ae0-b2d2-86fe8fe7ef00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx"
Sep 09 20:47:21.531375 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.531337    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b0cd0a-cf56-4821-915a-117f5b75556e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l\" (UID: \"63b0cd0a-cf56-4821-915a-117f5b75556e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l"
Sep 09 20:47:21.531443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.531373    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htlrk\" (UniqueName: \"kubernetes.io/projected/63b0cd0a-cf56-4821-915a-117f5b75556e-kube-api-access-htlrk\") pod \"payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l\" (UID: \"63b0cd0a-cf56-4821-915a-117f5b75556e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l"
Sep 09 20:47:21.531443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.531421    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/997cb48e-b246-4ae0-b2d2-86fe8fe7ef00-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx\" (UID: \"997cb48e-b246-4ae0-b2d2-86fe8fe7ef00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx"
Sep 09 20:47:21.532196 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.532148    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b0cd0a-cf56-4821-915a-117f5b75556e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l\" (UID: \"63b0cd0a-cf56-4821-915a-117f5b75556e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l"
Sep 09 20:47:21.541126 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.541104    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htlrk\" (UniqueName: \"kubernetes.io/projected/63b0cd0a-cf56-4821-915a-117f5b75556e-kube-api-access-htlrk\") pod \"payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l\" (UID: \"63b0cd0a-cf56-4821-915a-117f5b75556e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l"
Sep 09 20:47:21.632147 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.632060    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhbjc\" (UniqueName: \"kubernetes.io/projected/997cb48e-b246-4ae0-b2d2-86fe8fe7ef00-kube-api-access-jhbjc\") pod \"payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx\" (UID: \"997cb48e-b246-4ae0-b2d2-86fe8fe7ef00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx"
Sep 09 20:47:21.632147 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.632131    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/997cb48e-b246-4ae0-b2d2-86fe8fe7ef00-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx\" (UID: \"997cb48e-b246-4ae0-b2d2-86fe8fe7ef00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx"
Sep 09 20:47:21.633182 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.633085    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/997cb48e-b246-4ae0-b2d2-86fe8fe7ef00-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx\" (UID: \"997cb48e-b246-4ae0-b2d2-86fe8fe7ef00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx"
Sep 09 20:47:21.642156 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.642134    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhbjc\" (UniqueName: \"kubernetes.io/projected/997cb48e-b246-4ae0-b2d2-86fe8fe7ef00-kube-api-access-jhbjc\") pod \"payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx\" (UID: \"997cb48e-b246-4ae0-b2d2-86fe8fe7ef00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx"
Sep 09 20:47:21.756163 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.756137    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l"
Sep 09 20:47:21.816875 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.816840    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx"
Sep 09 20:47:21.927876 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.927807    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f122cb-jxc2s"
Sep 09 20:47:21.927876 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.927841    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f122cb-jxc2s" event={"ID":"4dbeed26-58e2-4d5c-a493-cbadebe3c484","Type":"ContainerDied","Data":"255405d8e52deb0542dce54b4c271cc239a4c072fc23924def87ba1704e2108b"}
Sep 09 20:47:21.928082 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.927883    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="255405d8e52deb0542dce54b4c271cc239a4c072fc23924def87ba1704e2108b"
Sep 09 20:47:21.931020 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.930832    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn" event={"ID":"59076416-48b5-44bf-93cd-15e75a5b6e60","Type":"ContainerStarted","Data":"273bff863f6db0a2a0e6050c26874d209a6bdc243358664d49a39897bda20b97"}
Sep 09 20:47:21.931020 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.930942    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn" event={"ID":"59076416-48b5-44bf-93cd-15e75a5b6e60","Type":"ContainerStarted","Data":"999798e97bef21d47d14cf61be37c3c3f5ad2f765c8db6b5c0e71356976bf4f1"}
Sep 09 20:47:21.931020 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.930974    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"
Sep 09 20:47:21.967468 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:21.967375    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn" podStartSLOduration=1.6253258659999998 podStartE2EDuration="1.96735482s" totalImagesPullingTime="342.028954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:47:21.963295002 +0000 UTC m=+887.643255311" watchObservedRunningTime="2026-09-09 20:47:21.96735482 +0000 UTC m=+887.647315129"
Sep 09 20:47:22.058233 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:22.058209    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l"]
Sep 09 20:47:22.060461 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:47:22.060423    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b0cd0a_cf56_4821_915a_117f5b75556e.slice/crio-73ce39c04c95a550bfc59494bbc8daf0563ef26b55c36d70c927e5f0aa15b35e WatchSource:0}: Error finding container 73ce39c04c95a550bfc59494bbc8daf0563ef26b55c36d70c927e5f0aa15b35e: Status 404 returned error can't find the container with id 73ce39c04c95a550bfc59494bbc8daf0563ef26b55c36d70c927e5f0aa15b35e
Sep 09 20:47:22.140826 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:22.140797    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx"]
Sep 09 20:47:22.143229 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:47:22.143188    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997cb48e_b246_4ae0_b2d2_86fe8fe7ef00.slice/crio-6baeddbe136bd65d9bcf94aefe0c7cd47afcdb327546426c6a344b32eaed9829 WatchSource:0}: Error finding container 6baeddbe136bd65d9bcf94aefe0c7cd47afcdb327546426c6a344b32eaed9829: Status 404 returned error can't find the container with id 6baeddbe136bd65d9bcf94aefe0c7cd47afcdb327546426c6a344b32eaed9829
Sep 09 20:47:22.406054 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:22.405748    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:47:22.406724 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:22.406651    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:47:22.407133 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:22.407117    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:47:22.937251 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:22.937217    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l" event={"ID":"63b0cd0a-cf56-4821-915a-117f5b75556e","Type":"ContainerStarted","Data":"55ab9816c069b0c6002549090fb7cde312658e49c584a01573c93f15fbd32b76"}
Sep 09 20:47:22.937413 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:22.937263    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l" event={"ID":"63b0cd0a-cf56-4821-915a-117f5b75556e","Type":"ContainerStarted","Data":"73ce39c04c95a550bfc59494bbc8daf0563ef26b55c36d70c927e5f0aa15b35e"}
Sep 09 20:47:22.937413 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:22.937329    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l"
Sep 09 20:47:22.938742 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:22.938718    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx" event={"ID":"997cb48e-b246-4ae0-b2d2-86fe8fe7ef00","Type":"ContainerStarted","Data":"1b720c7bd9c6966301a16834ab2b5e3fb5b0805a512483cf70c18bca2c8dbd92"}
Sep 09 20:47:22.938849 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:22.938761    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx" event={"ID":"997cb48e-b246-4ae0-b2d2-86fe8fe7ef00","Type":"ContainerStarted","Data":"6baeddbe136bd65d9bcf94aefe0c7cd47afcdb327546426c6a344b32eaed9829"}
Sep 09 20:47:22.939109 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:22.939086    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx"
Sep 09 20:47:22.956076 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:22.956032    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l" podStartSLOduration=1.956019653 podStartE2EDuration="1.956019653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:47:22.953964639 +0000 UTC m=+888.633924939" watchObservedRunningTime="2026-09-09 20:47:22.956019653 +0000 UTC m=+888.635979961"
Sep 09 20:47:22.973676 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:22.973628    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx" podStartSLOduration=1.973611237 podStartE2EDuration="1.973611237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:47:22.970484927 +0000 UTC m=+888.650445238" watchObservedRunningTime="2026-09-09 20:47:22.973611237 +0000 UTC m=+888.653571546"
Sep 09 20:47:25.568352 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:25.568306    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs"]
Sep 09 20:47:25.570010 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:25.568714    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs" podUID="78d3822b-e912-4127-9e4c-b3a6c18d296f" containerName="payload-pre-processing" containerID="cri-o://fcfaa517071f27ba863088d01c10647fed42cd57d54463508a1ad805a44a4ea2" gracePeriod=30
Sep 09 20:47:25.811594 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:25.811570    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs"
Sep 09 20:47:25.823816 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:25.823764    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f122cb-jxc2s"]
Sep 09 20:47:25.829901 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:25.829875    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f122cb-jxc2s"]
Sep 09 20:47:25.873859 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:25.873833    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d3822b-e912-4127-9e4c-b3a6c18d296f-plugins-config-volume\") pod \"78d3822b-e912-4127-9e4c-b3a6c18d296f\" (UID: \"78d3822b-e912-4127-9e4c-b3a6c18d296f\") "
Sep 09 20:47:25.874031 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:25.873865    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xlrt\" (UniqueName: \"kubernetes.io/projected/78d3822b-e912-4127-9e4c-b3a6c18d296f-kube-api-access-5xlrt\") pod \"78d3822b-e912-4127-9e4c-b3a6c18d296f\" (UID: \"78d3822b-e912-4127-9e4c-b3a6c18d296f\") "
Sep 09 20:47:25.874320 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:25.874291    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d3822b-e912-4127-9e4c-b3a6c18d296f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "78d3822b-e912-4127-9e4c-b3a6c18d296f" (UID: "78d3822b-e912-4127-9e4c-b3a6c18d296f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:47:25.876296 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:25.876270    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d3822b-e912-4127-9e4c-b3a6c18d296f-kube-api-access-5xlrt" (OuterVolumeSpecName: "kube-api-access-5xlrt") pod "78d3822b-e912-4127-9e4c-b3a6c18d296f" (UID: "78d3822b-e912-4127-9e4c-b3a6c18d296f"). InnerVolumeSpecName "kube-api-access-5xlrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:47:25.956291 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:25.956264    2556 generic.go:356] "Generic (PLEG): container finished" podID="78d3822b-e912-4127-9e4c-b3a6c18d296f" containerID="fcfaa517071f27ba863088d01c10647fed42cd57d54463508a1ad805a44a4ea2" exitCode=0
Sep 09 20:47:25.956440 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:25.956304    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs"
Sep 09 20:47:25.956440 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:25.956341    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs" event={"ID":"78d3822b-e912-4127-9e4c-b3a6c18d296f","Type":"ContainerDied","Data":"fcfaa517071f27ba863088d01c10647fed42cd57d54463508a1ad805a44a4ea2"}
Sep 09 20:47:25.956440 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:25.956394    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs" event={"ID":"78d3822b-e912-4127-9e4c-b3a6c18d296f","Type":"ContainerDied","Data":"e17169cd29cfe867da45df4c107cff5a50a329e76c7524bdab933af9dd78b27f"}
Sep 09 20:47:25.956440 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:25.956420    2556 scope.go:117] "RemoveContainer" containerID="fcfaa517071f27ba863088d01c10647fed42cd57d54463508a1ad805a44a4ea2"
Sep 09 20:47:25.965586 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:25.965571    2556 scope.go:117] "RemoveContainer" containerID="fcfaa517071f27ba863088d01c10647fed42cd57d54463508a1ad805a44a4ea2"
Sep 09 20:47:25.965858 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:47:25.965838    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcfaa517071f27ba863088d01c10647fed42cd57d54463508a1ad805a44a4ea2\": container with ID starting with fcfaa517071f27ba863088d01c10647fed42cd57d54463508a1ad805a44a4ea2 not found: ID does not exist" containerID="fcfaa517071f27ba863088d01c10647fed42cd57d54463508a1ad805a44a4ea2"
Sep 09 20:47:25.965938 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:25.965866    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcfaa517071f27ba863088d01c10647fed42cd57d54463508a1ad805a44a4ea2"} err="failed to get container status \"fcfaa517071f27ba863088d01c10647fed42cd57d54463508a1ad805a44a4ea2\": rpc error: code = NotFound desc = could not find container \"fcfaa517071f27ba863088d01c10647fed42cd57d54463508a1ad805a44a4ea2\": container with ID starting with fcfaa517071f27ba863088d01c10647fed42cd57d54463508a1ad805a44a4ea2 not found: ID does not exist"
Sep 09 20:47:25.975082 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:25.975059    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d3822b-e912-4127-9e4c-b3a6c18d296f-plugins-config-volume\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:47:25.975184 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:25.975084    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xlrt\" (UniqueName: \"kubernetes.io/projected/78d3822b-e912-4127-9e4c-b3a6c18d296f-kube-api-access-5xlrt\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:47:25.986892 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:25.986870    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs"]
Sep 09 20:47:25.993605 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:25.993585    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04f122cb-c7757c6bb-4pbjs"]
Sep 09 20:47:26.892401 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:26.892361    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dbeed26-58e2-4d5c-a493-cbadebe3c484" path="/var/lib/kubelet/pods/4dbeed26-58e2-4d5c-a493-cbadebe3c484/volumes"
Sep 09 20:47:26.892897 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:26.892878    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78d3822b-e912-4127-9e4c-b3a6c18d296f" path="/var/lib/kubelet/pods/78d3822b-e912-4127-9e4c-b3a6c18d296f/volumes"
Sep 09 20:47:27.943880 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:27.943802    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"
Sep 09 20:47:31.320820 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:31.320784    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm"]
Sep 09 20:47:31.321265 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:31.321251    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78d3822b-e912-4127-9e4c-b3a6c18d296f" containerName="payload-pre-processing"
Sep 09 20:47:31.321265 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:31.321266    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="78d3822b-e912-4127-9e4c-b3a6c18d296f" containerName="payload-pre-processing"
Sep 09 20:47:31.321353 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:31.321327    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="78d3822b-e912-4127-9e4c-b3a6c18d296f" containerName="payload-pre-processing"
Sep 09 20:47:31.323987 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:31.323972    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm"
Sep 09 20:47:31.369249 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:31.369222    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-5333ad9e-dockercfg-zkqkq\""
Sep 09 20:47:31.370414 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:31.370391    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-5333ad9e\""
Sep 09 20:47:31.392167 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:31.392143    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm"]
Sep 09 20:47:31.429753 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:31.429724    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68fd68d0-1c9f-4f05-b25e-d4fac37e01a9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm\" (UID: \"68fd68d0-1c9f-4f05-b25e-d4fac37e01a9\") " pod="openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm"
Sep 09 20:47:31.429928 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:31.429779    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq7h7\" (UniqueName: \"kubernetes.io/projected/68fd68d0-1c9f-4f05-b25e-d4fac37e01a9-kube-api-access-zq7h7\") pod \"payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm\" (UID: \"68fd68d0-1c9f-4f05-b25e-d4fac37e01a9\") " pod="openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm"
Sep 09 20:47:31.530833 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:31.530803    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68fd68d0-1c9f-4f05-b25e-d4fac37e01a9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm\" (UID: \"68fd68d0-1c9f-4f05-b25e-d4fac37e01a9\") " pod="openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm"
Sep 09 20:47:31.531005 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:31.530847    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq7h7\" (UniqueName: \"kubernetes.io/projected/68fd68d0-1c9f-4f05-b25e-d4fac37e01a9-kube-api-access-zq7h7\") pod \"payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm\" (UID: \"68fd68d0-1c9f-4f05-b25e-d4fac37e01a9\") " pod="openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm"
Sep 09 20:47:31.531584 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:31.531565    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68fd68d0-1c9f-4f05-b25e-d4fac37e01a9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm\" (UID: \"68fd68d0-1c9f-4f05-b25e-d4fac37e01a9\") " pod="openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm"
Sep 09 20:47:31.543331 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:31.543302    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:47:31.561942 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:31.561919    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq7h7\" (UniqueName: \"kubernetes.io/projected/68fd68d0-1c9f-4f05-b25e-d4fac37e01a9-kube-api-access-zq7h7\") pod \"payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm\" (UID: \"68fd68d0-1c9f-4f05-b25e-d4fac37e01a9\") " pod="openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm"
Sep 09 20:47:31.634464 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:31.634411    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm"
Sep 09 20:47:31.920103 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:31.920079    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm"]
Sep 09 20:47:31.921781 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:47:31.921756    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68fd68d0_1c9f_4f05_b25e_d4fac37e01a9.slice/crio-0f31071b35b8a43f56cb5518ba1adee33a94c697281d9c6f08b1dab9b0bca4ad WatchSource:0}: Error finding container 0f31071b35b8a43f56cb5518ba1adee33a94c697281d9c6f08b1dab9b0bca4ad: Status 404 returned error can't find the container with id 0f31071b35b8a43f56cb5518ba1adee33a94c697281d9c6f08b1dab9b0bca4ad
Sep 09 20:47:31.989948 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:31.989924    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm" event={"ID":"68fd68d0-1c9f-4f05-b25e-d4fac37e01a9","Type":"ContainerStarted","Data":"0f31071b35b8a43f56cb5518ba1adee33a94c697281d9c6f08b1dab9b0bca4ad"}
Sep 09 20:47:32.997173 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:32.997138    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm" event={"ID":"68fd68d0-1c9f-4f05-b25e-d4fac37e01a9","Type":"ContainerStarted","Data":"5b046a9b843e8642dde9bca5d2627dce90d7dc139ded044c397a653cc9862750"}
Sep 09 20:47:32.997598 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:32.997266    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm"
Sep 09 20:47:33.104586 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:33.104510    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm" podStartSLOduration=2.104485803 podStartE2EDuration="2.104485803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:47:33.100238697 +0000 UTC m=+898.780199007" watchObservedRunningTime="2026-09-09 20:47:33.104485803 +0000 UTC m=+898.784446112"
Sep 09 20:47:33.947005 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:33.946977    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l"
Sep 09 20:47:34.952074 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:34.952043    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx"
Sep 09 20:47:42.051588 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:42.051552    2556 generic.go:356] "Generic (PLEG): container finished" podID="fbbd858d-84a8-40c5-a9ea-036ee3597316" containerID="8083f01b6fa208c0eb6e67037b8ad02e2cfcc6d0cd3daafc03f01df602776080" exitCode=0
Sep 09 20:47:42.052004 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:42.051623    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz" event={"ID":"fbbd858d-84a8-40c5-a9ea-036ee3597316","Type":"ContainerDied","Data":"8083f01b6fa208c0eb6e67037b8ad02e2cfcc6d0cd3daafc03f01df602776080"}
Sep 09 20:47:42.052114 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:42.052096    2556 scope.go:117] "RemoveContainer" containerID="8083f01b6fa208c0eb6e67037b8ad02e2cfcc6d0cd3daafc03f01df602776080"
Sep 09 20:47:42.053179 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:42.053153    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:47:42.053249 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:42.053224    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:47:42.053291 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:42.053256    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:47:43.059205 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:43.059173    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz" event={"ID":"fbbd858d-84a8-40c5-a9ea-036ee3597316","Type":"ContainerStarted","Data":"2c14522b69d13824c6c21b9e4e0deb530a15a72e57874f96cfda92591e18e083"}
Sep 09 20:47:43.187707 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:43.187667    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:43.187928 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:43.187792    2556 patch_prober.go:28] interesting pod/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 09 20:47:43.187928 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:43.187836    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz" podUID="fbbd858d-84a8-40c5-a9ea-036ee3597316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 09 20:47:44.006065 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:44.006038    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm"
Sep 09 20:47:44.188494 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:44.188073    2556 patch_prober.go:28] interesting pod/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 09 20:47:44.188494 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:44.188128    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz" podUID="fbbd858d-84a8-40c5-a9ea-036ee3597316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 09 20:47:45.188245 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:45.188206    2556 patch_prober.go:28] interesting pod/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 09 20:47:45.188447 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:45.188269    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz" podUID="fbbd858d-84a8-40c5-a9ea-036ee3597316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 09 20:47:46.187432 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:46.187398    2556 patch_prober.go:28] interesting pod/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 09 20:47:46.188136 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:46.188105    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz" podUID="fbbd858d-84a8-40c5-a9ea-036ee3597316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 09 20:47:47.188022 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:47.187981    2556 patch_prober.go:28] interesting pod/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 09 20:47:47.188518 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:47.188059    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz" podUID="fbbd858d-84a8-40c5-a9ea-036ee3597316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 09 20:47:47.899086 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:47.899051    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:48.191782 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:48.191696    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:48.192758 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:48.192742    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:47:49.735869 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:47:49.735834    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:48:01.700843 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:01.700812    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm"]
Sep 09 20:48:01.701307 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:01.701174    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm" podUID="68fd68d0-1c9f-4f05-b25e-d4fac37e01a9" containerName="payload-processing" containerID="cri-o://5b046a9b843e8642dde9bca5d2627dce90d7dc139ded044c397a653cc9862750" gracePeriod=30
Sep 09 20:48:01.949099 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:01.949073    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm"
Sep 09 20:48:02.040945 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:02.040890    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68fd68d0-1c9f-4f05-b25e-d4fac37e01a9-plugins-config-volume\") pod \"68fd68d0-1c9f-4f05-b25e-d4fac37e01a9\" (UID: \"68fd68d0-1c9f-4f05-b25e-d4fac37e01a9\") "
Sep 09 20:48:02.041096 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:02.040988    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq7h7\" (UniqueName: \"kubernetes.io/projected/68fd68d0-1c9f-4f05-b25e-d4fac37e01a9-kube-api-access-zq7h7\") pod \"68fd68d0-1c9f-4f05-b25e-d4fac37e01a9\" (UID: \"68fd68d0-1c9f-4f05-b25e-d4fac37e01a9\") "
Sep 09 20:48:02.041257 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:02.041234    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68fd68d0-1c9f-4f05-b25e-d4fac37e01a9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "68fd68d0-1c9f-4f05-b25e-d4fac37e01a9" (UID: "68fd68d0-1c9f-4f05-b25e-d4fac37e01a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:48:02.042872 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:02.042845    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68fd68d0-1c9f-4f05-b25e-d4fac37e01a9-kube-api-access-zq7h7" (OuterVolumeSpecName: "kube-api-access-zq7h7") pod "68fd68d0-1c9f-4f05-b25e-d4fac37e01a9" (UID: "68fd68d0-1c9f-4f05-b25e-d4fac37e01a9"). InnerVolumeSpecName "kube-api-access-zq7h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:48:02.142371 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:02.142338    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68fd68d0-1c9f-4f05-b25e-d4fac37e01a9-plugins-config-volume\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:48:02.142371 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:02.142367    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq7h7\" (UniqueName: \"kubernetes.io/projected/68fd68d0-1c9f-4f05-b25e-d4fac37e01a9-kube-api-access-zq7h7\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:48:02.170615 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:02.170587    2556 generic.go:356] "Generic (PLEG): container finished" podID="68fd68d0-1c9f-4f05-b25e-d4fac37e01a9" containerID="5b046a9b843e8642dde9bca5d2627dce90d7dc139ded044c397a653cc9862750" exitCode=0
Sep 09 20:48:02.170730 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:02.170625    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm"
Sep 09 20:48:02.170730 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:02.170630    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm" event={"ID":"68fd68d0-1c9f-4f05-b25e-d4fac37e01a9","Type":"ContainerDied","Data":"5b046a9b843e8642dde9bca5d2627dce90d7dc139ded044c397a653cc9862750"}
Sep 09 20:48:02.170730 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:02.170684    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm" event={"ID":"68fd68d0-1c9f-4f05-b25e-d4fac37e01a9","Type":"ContainerDied","Data":"0f31071b35b8a43f56cb5518ba1adee33a94c697281d9c6f08b1dab9b0bca4ad"}
Sep 09 20:48:02.170730 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:02.170706    2556 scope.go:117] "RemoveContainer" containerID="5b046a9b843e8642dde9bca5d2627dce90d7dc139ded044c397a653cc9862750"
Sep 09 20:48:02.179959 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:02.179942    2556 scope.go:117] "RemoveContainer" containerID="5b046a9b843e8642dde9bca5d2627dce90d7dc139ded044c397a653cc9862750"
Sep 09 20:48:02.180210 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:48:02.180193    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b046a9b843e8642dde9bca5d2627dce90d7dc139ded044c397a653cc9862750\": container with ID starting with 5b046a9b843e8642dde9bca5d2627dce90d7dc139ded044c397a653cc9862750 not found: ID does not exist" containerID="5b046a9b843e8642dde9bca5d2627dce90d7dc139ded044c397a653cc9862750"
Sep 09 20:48:02.180263 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:02.180218    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b046a9b843e8642dde9bca5d2627dce90d7dc139ded044c397a653cc9862750"} err="failed to get container status \"5b046a9b843e8642dde9bca5d2627dce90d7dc139ded044c397a653cc9862750\": rpc error: code = NotFound desc = could not find container \"5b046a9b843e8642dde9bca5d2627dce90d7dc139ded044c397a653cc9862750\": container with ID starting with 5b046a9b843e8642dde9bca5d2627dce90d7dc139ded044c397a653cc9862750 not found: ID does not exist"
Sep 09 20:48:02.194314 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:02.194290    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm"]
Sep 09 20:48:02.198947 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:02.198926    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5333ad9e-7b966f877d-sr6tm"]
Sep 09 20:48:02.895494 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:02.895465    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68fd68d0-1c9f-4f05-b25e-d4fac37e01a9" path="/var/lib/kubelet/pods/68fd68d0-1c9f-4f05-b25e-d4fac37e01a9/volumes"
Sep 09 20:48:06.845324 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:06.845285    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9"]
Sep 09 20:48:06.845758 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:06.845743    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68fd68d0-1c9f-4f05-b25e-d4fac37e01a9" containerName="payload-processing"
Sep 09 20:48:06.845813 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:06.845760    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="68fd68d0-1c9f-4f05-b25e-d4fac37e01a9" containerName="payload-processing"
Sep 09 20:48:06.845851 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:06.845844    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="68fd68d0-1c9f-4f05-b25e-d4fac37e01a9" containerName="payload-processing"
Sep 09 20:48:06.848503 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:06.848487    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9"
Sep 09 20:48:06.851366 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:06.851339    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-fd5982ad\""
Sep 09 20:48:06.851483 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:06.851347    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-fd5982ad-dockercfg-rbdh7\""
Sep 09 20:48:06.862473 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:06.862447    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9"]
Sep 09 20:48:06.984824 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:06.984792    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd4sq\" (UniqueName: \"kubernetes.io/projected/a8bf8042-b34d-4e0d-bc12-d0b779fa6aea-kube-api-access-bd4sq\") pod \"payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9\" (UID: \"a8bf8042-b34d-4e0d-bc12-d0b779fa6aea\") " pod="openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9"
Sep 09 20:48:06.985005 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:06.984834    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8bf8042-b34d-4e0d-bc12-d0b779fa6aea-plugins-config-volume\") pod \"payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9\" (UID: \"a8bf8042-b34d-4e0d-bc12-d0b779fa6aea\") " pod="openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9"
Sep 09 20:48:07.086335 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:07.086306    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd4sq\" (UniqueName: \"kubernetes.io/projected/a8bf8042-b34d-4e0d-bc12-d0b779fa6aea-kube-api-access-bd4sq\") pod \"payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9\" (UID: \"a8bf8042-b34d-4e0d-bc12-d0b779fa6aea\") " pod="openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9"
Sep 09 20:48:07.086496 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:07.086342    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8bf8042-b34d-4e0d-bc12-d0b779fa6aea-plugins-config-volume\") pod \"payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9\" (UID: \"a8bf8042-b34d-4e0d-bc12-d0b779fa6aea\") " pod="openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9"
Sep 09 20:48:07.086965 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:07.086946    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8bf8042-b34d-4e0d-bc12-d0b779fa6aea-plugins-config-volume\") pod \"payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9\" (UID: \"a8bf8042-b34d-4e0d-bc12-d0b779fa6aea\") " pod="openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9"
Sep 09 20:48:07.095636 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:07.095572    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd4sq\" (UniqueName: \"kubernetes.io/projected/a8bf8042-b34d-4e0d-bc12-d0b779fa6aea-kube-api-access-bd4sq\") pod \"payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9\" (UID: \"a8bf8042-b34d-4e0d-bc12-d0b779fa6aea\") " pod="openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9"
Sep 09 20:48:07.158956 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:07.158931    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9"
Sep 09 20:48:07.436494 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:07.436467    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9"]
Sep 09 20:48:07.438835 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:48:07.438795    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bf8042_b34d_4e0d_bc12_d0b779fa6aea.slice/crio-9d17e6e5ab3fbfacc306db7f7d5596be2f6cb50defdd245da105f32c3d38d145 WatchSource:0}: Error finding container 9d17e6e5ab3fbfacc306db7f7d5596be2f6cb50defdd245da105f32c3d38d145: Status 404 returned error can't find the container with id 9d17e6e5ab3fbfacc306db7f7d5596be2f6cb50defdd245da105f32c3d38d145
Sep 09 20:48:08.209576 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:08.209541    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9" event={"ID":"a8bf8042-b34d-4e0d-bc12-d0b779fa6aea","Type":"ContainerStarted","Data":"418788c8471f7c82d59d4404f5fe161f3ab6f33aa56b017aeb66eac88e51ac40"}
Sep 09 20:48:08.209979 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:08.209613    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9" event={"ID":"a8bf8042-b34d-4e0d-bc12-d0b779fa6aea","Type":"ContainerStarted","Data":"9d17e6e5ab3fbfacc306db7f7d5596be2f6cb50defdd245da105f32c3d38d145"}
Sep 09 20:48:08.209979 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:08.209658    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9"
Sep 09 20:48:08.228860 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:08.228817    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9" podStartSLOduration=2.228804317 podStartE2EDuration="2.228804317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:08.226660011 +0000 UTC m=+933.906620322" watchObservedRunningTime="2026-09-09 20:48:08.228804317 +0000 UTC m=+933.908764626"
Sep 09 20:48:11.623244 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:11.623208    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:48:19.217502 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:19.217474    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9"
Sep 09 20:48:24.146635 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.146601    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"]
Sep 09 20:48:24.151536 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.151514    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:24.159676 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.159658    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-2beba0\"/\"kube-root-ca.crt\""
Sep 09 20:48:24.159860 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.159838    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-2beba0\"/\"default-dockercfg-j9g9v\""
Sep 09 20:48:24.161121 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.161106    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-2beba0\"/\"autores-model-2beba0-kserve-self-signed-certs\""
Sep 09 20:48:24.162578 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.162564    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-2beba0\"/\"openshift-service-ca.crt\""
Sep 09 20:48:24.176976 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.176950    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"]
Sep 09 20:48:24.244407 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.244383    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-dshm\") pod \"autores-model-2beba0-kserve-6d6684b655-49cmh\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") " pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:24.244532 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.244420    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpp2c\" (UniqueName: \"kubernetes.io/projected/60b4490b-6b81-4410-a185-27654a7227dd-kube-api-access-mpp2c\") pod \"autores-model-2beba0-kserve-6d6684b655-49cmh\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") " pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:24.244532 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.244446    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-tmp-dir\") pod \"autores-model-2beba0-kserve-6d6684b655-49cmh\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") " pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:24.244532 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.244519    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-model-cache\") pod \"autores-model-2beba0-kserve-6d6684b655-49cmh\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") " pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:24.244647 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.244538    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-home\") pod \"autores-model-2beba0-kserve-6d6684b655-49cmh\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") " pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:24.244647 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.244599    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b4490b-6b81-4410-a185-27654a7227dd-tls-certs\") pod \"autores-model-2beba0-kserve-6d6684b655-49cmh\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") " pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:24.345221 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.345193    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-dshm\") pod \"autores-model-2beba0-kserve-6d6684b655-49cmh\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") " pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:24.345366 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.345232    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpp2c\" (UniqueName: \"kubernetes.io/projected/60b4490b-6b81-4410-a185-27654a7227dd-kube-api-access-mpp2c\") pod \"autores-model-2beba0-kserve-6d6684b655-49cmh\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") " pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:24.345366 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.345265    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-tmp-dir\") pod \"autores-model-2beba0-kserve-6d6684b655-49cmh\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") " pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:24.345366 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.345311    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-model-cache\") pod \"autores-model-2beba0-kserve-6d6684b655-49cmh\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") " pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:24.345366 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.345335    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-home\") pod \"autores-model-2beba0-kserve-6d6684b655-49cmh\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") " pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:24.345366 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.345354    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b4490b-6b81-4410-a185-27654a7227dd-tls-certs\") pod \"autores-model-2beba0-kserve-6d6684b655-49cmh\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") " pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:24.345715 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.345692    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-tmp-dir\") pod \"autores-model-2beba0-kserve-6d6684b655-49cmh\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") " pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:24.345782 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.345709    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-model-cache\") pod \"autores-model-2beba0-kserve-6d6684b655-49cmh\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") " pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:24.345782 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.345760    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-home\") pod \"autores-model-2beba0-kserve-6d6684b655-49cmh\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") " pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:24.347431 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.347414    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-dshm\") pod \"autores-model-2beba0-kserve-6d6684b655-49cmh\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") " pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:24.347704 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.347684    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b4490b-6b81-4410-a185-27654a7227dd-tls-certs\") pod \"autores-model-2beba0-kserve-6d6684b655-49cmh\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") " pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:24.355403 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.355380    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpp2c\" (UniqueName: \"kubernetes.io/projected/60b4490b-6b81-4410-a185-27654a7227dd-kube-api-access-mpp2c\") pod \"autores-model-2beba0-kserve-6d6684b655-49cmh\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") " pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:24.460401 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.460351    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:24.740877 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:24.740847    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"]
Sep 09 20:48:24.741489 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:48:24.741466    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60b4490b_6b81_4410_a185_27654a7227dd.slice/crio-8dc9dc840228c6e21847e16ecc24257d9a7b45bae48674bee952daee8b8e2572 WatchSource:0}: Error finding container 8dc9dc840228c6e21847e16ecc24257d9a7b45bae48674bee952daee8b8e2572: Status 404 returned error can't find the container with id 8dc9dc840228c6e21847e16ecc24257d9a7b45bae48674bee952daee8b8e2572
Sep 09 20:48:25.301710 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:25.301675    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh" event={"ID":"60b4490b-6b81-4410-a185-27654a7227dd","Type":"ContainerStarted","Data":"be3c71b5f331180d26bb27d32262ebb9686fd9bf626f908af5b7e4ff0ccb3ff8"}
Sep 09 20:48:25.302109 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:25.301718    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh" event={"ID":"60b4490b-6b81-4410-a185-27654a7227dd","Type":"ContainerStarted","Data":"8dc9dc840228c6e21847e16ecc24257d9a7b45bae48674bee952daee8b8e2572"}
Sep 09 20:48:25.323503 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:25.323456    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh" podStartSLOduration=1.32344248 podStartE2EDuration="1.32344248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:25.321458734 +0000 UTC m=+951.001419045" watchObservedRunningTime="2026-09-09 20:48:25.32344248 +0000 UTC m=+951.003402789"
Sep 09 20:48:33.410702 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:33.410668    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:48:34.460883 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:34.460849    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:34.460883 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:34.460890    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:34.473188 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:34.473168    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:35.367254 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:35.367226    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:48:36.898949 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:36.898917    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9"]
Sep 09 20:48:36.899374 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:36.899158    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9" podUID="a8bf8042-b34d-4e0d-bc12-d0b779fa6aea" containerName="payload-processing" containerID="cri-o://418788c8471f7c82d59d4404f5fe161f3ab6f33aa56b017aeb66eac88e51ac40" gracePeriod=30
Sep 09 20:48:36.963075 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:48:36.963033    2556 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bf8042_b34d_4e0d_bc12_d0b779fa6aea.slice/crio-conmon-418788c8471f7c82d59d4404f5fe161f3ab6f33aa56b017aeb66eac88e51ac40.scope\": RecentStats: unable to find data in memory cache]"
Sep 09 20:48:37.140265 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:37.140243    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9"
Sep 09 20:48:37.163365 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:37.163290    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8bf8042-b34d-4e0d-bc12-d0b779fa6aea-plugins-config-volume\") pod \"a8bf8042-b34d-4e0d-bc12-d0b779fa6aea\" (UID: \"a8bf8042-b34d-4e0d-bc12-d0b779fa6aea\") "
Sep 09 20:48:37.163365 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:37.163361    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bd4sq\" (UniqueName: \"kubernetes.io/projected/a8bf8042-b34d-4e0d-bc12-d0b779fa6aea-kube-api-access-bd4sq\") pod \"a8bf8042-b34d-4e0d-bc12-d0b779fa6aea\" (UID: \"a8bf8042-b34d-4e0d-bc12-d0b779fa6aea\") "
Sep 09 20:48:37.163717 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:37.163691    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8bf8042-b34d-4e0d-bc12-d0b779fa6aea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a8bf8042-b34d-4e0d-bc12-d0b779fa6aea" (UID: "a8bf8042-b34d-4e0d-bc12-d0b779fa6aea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:48:37.165579 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:37.165547    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8bf8042-b34d-4e0d-bc12-d0b779fa6aea-kube-api-access-bd4sq" (OuterVolumeSpecName: "kube-api-access-bd4sq") pod "a8bf8042-b34d-4e0d-bc12-d0b779fa6aea" (UID: "a8bf8042-b34d-4e0d-bc12-d0b779fa6aea"). InnerVolumeSpecName "kube-api-access-bd4sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:48:37.265187 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:37.265154    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8bf8042-b34d-4e0d-bc12-d0b779fa6aea-plugins-config-volume\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:48:37.265187 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:37.265182    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd4sq\" (UniqueName: \"kubernetes.io/projected/a8bf8042-b34d-4e0d-bc12-d0b779fa6aea-kube-api-access-bd4sq\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:48:37.368539 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:37.368508    2556 generic.go:356] "Generic (PLEG): container finished" podID="a8bf8042-b34d-4e0d-bc12-d0b779fa6aea" containerID="418788c8471f7c82d59d4404f5fe161f3ab6f33aa56b017aeb66eac88e51ac40" exitCode=0
Sep 09 20:48:37.368681 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:37.368554    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9"
Sep 09 20:48:37.368681 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:37.368577    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9" event={"ID":"a8bf8042-b34d-4e0d-bc12-d0b779fa6aea","Type":"ContainerDied","Data":"418788c8471f7c82d59d4404f5fe161f3ab6f33aa56b017aeb66eac88e51ac40"}
Sep 09 20:48:37.368681 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:37.368644    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9" event={"ID":"a8bf8042-b34d-4e0d-bc12-d0b779fa6aea","Type":"ContainerDied","Data":"9d17e6e5ab3fbfacc306db7f7d5596be2f6cb50defdd245da105f32c3d38d145"}
Sep 09 20:48:37.368681 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:37.368666    2556 scope.go:117] "RemoveContainer" containerID="418788c8471f7c82d59d4404f5fe161f3ab6f33aa56b017aeb66eac88e51ac40"
Sep 09 20:48:37.377997 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:37.377979    2556 scope.go:117] "RemoveContainer" containerID="418788c8471f7c82d59d4404f5fe161f3ab6f33aa56b017aeb66eac88e51ac40"
Sep 09 20:48:37.378262 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:48:37.378245    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"418788c8471f7c82d59d4404f5fe161f3ab6f33aa56b017aeb66eac88e51ac40\": container with ID starting with 418788c8471f7c82d59d4404f5fe161f3ab6f33aa56b017aeb66eac88e51ac40 not found: ID does not exist" containerID="418788c8471f7c82d59d4404f5fe161f3ab6f33aa56b017aeb66eac88e51ac40"
Sep 09 20:48:37.378314 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:37.378270    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"418788c8471f7c82d59d4404f5fe161f3ab6f33aa56b017aeb66eac88e51ac40"} err="failed to get container status \"418788c8471f7c82d59d4404f5fe161f3ab6f33aa56b017aeb66eac88e51ac40\": rpc error: code = NotFound desc = could not find container \"418788c8471f7c82d59d4404f5fe161f3ab6f33aa56b017aeb66eac88e51ac40\": container with ID starting with 418788c8471f7c82d59d4404f5fe161f3ab6f33aa56b017aeb66eac88e51ac40 not found: ID does not exist"
Sep 09 20:48:37.390425 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:37.390397    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9"]
Sep 09 20:48:37.394511 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:37.394491    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fd5982ad-b68c4858f-rxzd9"]
Sep 09 20:48:38.892534 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:38.892500    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8bf8042-b34d-4e0d-bc12-d0b779fa6aea" path="/var/lib/kubelet/pods/a8bf8042-b34d-4e0d-bc12-d0b779fa6aea/volumes"
Sep 09 20:48:38.920883 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:38.920844    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"]
Sep 09 20:48:38.921554 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:38.921529    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8bf8042-b34d-4e0d-bc12-d0b779fa6aea" containerName="payload-processing"
Sep 09 20:48:38.921554 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:38.921553    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8bf8042-b34d-4e0d-bc12-d0b779fa6aea" containerName="payload-processing"
Sep 09 20:48:38.921746 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:38.921656    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8bf8042-b34d-4e0d-bc12-d0b779fa6aea" containerName="payload-processing"
Sep 09 20:48:38.926994 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:38.926896    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:38.930572 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:38.930551    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-3d3c9036-openshift-default-dockercfg-6mqv2\""
Sep 09 20:48:38.941411 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:38.941389    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"]
Sep 09 20:48:38.980485 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:38.980453    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-token\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:38.980617 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:38.980494    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-workload-socket\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:38.980617 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:38.980515    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-data\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:38.980736 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:38.980632    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fsb9\" (UniqueName: \"kubernetes.io/projected/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-kube-api-access-8fsb9\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:38.980736 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:38.980668    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istiod-ca-cert\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:38.980736 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:38.980725    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-podinfo\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:38.980855 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:38.980744    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-workload-certs\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:38.980855 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:38.980757    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-envoy\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:38.980855 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:38.980816    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-credential-socket\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:39.081536 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:39.081514    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-credential-socket\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:39.081722 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:39.081574    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-token\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:39.081722 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:39.081597    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-workload-socket\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:39.081722 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:39.081624    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-data\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:39.081941 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:39.081742    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fsb9\" (UniqueName: \"kubernetes.io/projected/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-kube-api-access-8fsb9\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:39.081941 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:39.081783    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istiod-ca-cert\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:39.081941 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:39.081825    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-podinfo\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:39.081941 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:39.081853    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-workload-certs\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:39.082189 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:39.081954    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-credential-socket\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:39.082189 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:39.081958    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-envoy\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:39.082189 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:39.081994    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-workload-socket\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:39.082189 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:39.082151    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-workload-certs\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:39.082189 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:39.082184    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-data\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:39.082537 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:39.082516    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istiod-ca-cert\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:39.084060 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:39.084040    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-envoy\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:39.084249 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:39.084229    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-podinfo\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:39.091657 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:39.091636    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-token\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:39.091893 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:39.091875    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fsb9\" (UniqueName: \"kubernetes.io/projected/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-kube-api-access-8fsb9\") pod \"e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") " pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:39.239533 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:39.239503    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:39.518192 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:39.518167    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"]
Sep 09 20:48:39.519109 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:48:39.519082    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd870d634_d19c_4b9b_b5cf_85ca2eeae4f5.slice/crio-b98c48d14492f3f59fdfab36a3a2a4ea9df3cac4983e38589512fd352b5db440 WatchSource:0}: Error finding container b98c48d14492f3f59fdfab36a3a2a4ea9df3cac4983e38589512fd352b5db440: Status 404 returned error can't find the container with id b98c48d14492f3f59fdfab36a3a2a4ea9df3cac4983e38589512fd352b5db440
Sep 09 20:48:39.521448 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:39.521415    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:48:39.521532 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:39.521479    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:48:39.521532 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:39.521507    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:48:40.390026 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:40.389990    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r" event={"ID":"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5","Type":"ContainerStarted","Data":"490d7d6d394b3595c51e824c359ce72e0af1aca24f76f3672990e265ff61f4c6"}
Sep 09 20:48:40.390403 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:40.390043    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r" event={"ID":"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5","Type":"ContainerStarted","Data":"b98c48d14492f3f59fdfab36a3a2a4ea9df3cac4983e38589512fd352b5db440"}
Sep 09 20:48:40.431039 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:40.430990    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r" podStartSLOduration=2.430973333 podStartE2EDuration="2.430973333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:40.427608132 +0000 UTC m=+966.107568440" watchObservedRunningTime="2026-09-09 20:48:40.430973333 +0000 UTC m=+966.110933642"
Sep 09 20:48:41.239961 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:41.239927    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:41.245249 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:41.245223    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:41.395298 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:41.395266    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:41.396165 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:41.396146    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:48:54.704225 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:48:54.704189    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:49:05.534922 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:05.534876    2556 generic.go:356] "Generic (PLEG): container finished" podID="d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" containerID="490d7d6d394b3595c51e824c359ce72e0af1aca24f76f3672990e265ff61f4c6" exitCode=0
Sep 09 20:49:05.535314 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:05.534930    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r" event={"ID":"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5","Type":"ContainerDied","Data":"490d7d6d394b3595c51e824c359ce72e0af1aca24f76f3672990e265ff61f4c6"}
Sep 09 20:49:05.535364 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:05.535315    2556 scope.go:117] "RemoveContainer" containerID="490d7d6d394b3595c51e824c359ce72e0af1aca24f76f3672990e265ff61f4c6"
Sep 09 20:49:05.536437 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:05.536407    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:49:05.536496 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:05.536484    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:49:05.536534 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:05.536518    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:49:06.542717 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:06.542686    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r" event={"ID":"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5","Type":"ContainerStarted","Data":"0997d8e6c0a34fee496f9f25239d511bac84ad5acec869266671168839a4217e"}
Sep 09 20:49:07.239847 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:07.239816    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:49:07.240893 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:07.240866    2556 patch_prober.go:28] interesting pod/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 09 20:49:07.241031 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:07.240928    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r" podUID="d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 09 20:49:08.240201 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:08.240168    2556 patch_prober.go:28] interesting pod/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 09 20:49:08.240591 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:08.240232    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r" podUID="d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 09 20:49:09.240217 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:09.240179    2556 patch_prober.go:28] interesting pod/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 09 20:49:09.240640 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:09.240237    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r" podUID="d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 09 20:49:10.100181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:10.100147    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5"]
Sep 09 20:49:10.100648 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:10.100616    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5" podUID="2caa6df3-5b63-4212-b03c-8909bb2f4f92" containerName="payload-pre-processing" containerID="cri-o://3b43e54a0f1dbc43d34bdfb6e1b2fbf2a483ca715b088a0e83ffaacc9ffb7822" gracePeriod=30
Sep 09 20:49:10.240386 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:10.240352    2556 patch_prober.go:28] interesting pod/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 09 20:49:10.240731 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:10.240407    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r" podUID="d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 09 20:49:10.391100 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:10.390808    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5"
Sep 09 20:49:10.397412 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:10.396581    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2caa6df3-5b63-4212-b03c-8909bb2f4f92-plugins-config-volume\") pod \"2caa6df3-5b63-4212-b03c-8909bb2f4f92\" (UID: \"2caa6df3-5b63-4212-b03c-8909bb2f4f92\") "
Sep 09 20:49:10.397412 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:10.396645    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67skl\" (UniqueName: \"kubernetes.io/projected/2caa6df3-5b63-4212-b03c-8909bb2f4f92-kube-api-access-67skl\") pod \"2caa6df3-5b63-4212-b03c-8909bb2f4f92\" (UID: \"2caa6df3-5b63-4212-b03c-8909bb2f4f92\") "
Sep 09 20:49:10.400379 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:10.400333    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2caa6df3-5b63-4212-b03c-8909bb2f4f92-kube-api-access-67skl" (OuterVolumeSpecName: "kube-api-access-67skl") pod "2caa6df3-5b63-4212-b03c-8909bb2f4f92" (UID: "2caa6df3-5b63-4212-b03c-8909bb2f4f92"). InnerVolumeSpecName "kube-api-access-67skl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:49:10.405930 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:10.404093    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2caa6df3-5b63-4212-b03c-8909bb2f4f92-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2caa6df3-5b63-4212-b03c-8909bb2f4f92" (UID: "2caa6df3-5b63-4212-b03c-8909bb2f4f92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:49:10.497949 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:10.497894    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2caa6df3-5b63-4212-b03c-8909bb2f4f92-plugins-config-volume\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:10.497949 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:10.497950    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67skl\" (UniqueName: \"kubernetes.io/projected/2caa6df3-5b63-4212-b03c-8909bb2f4f92-kube-api-access-67skl\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:10.565245 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:10.565215    2556 generic.go:356] "Generic (PLEG): container finished" podID="2caa6df3-5b63-4212-b03c-8909bb2f4f92" containerID="3b43e54a0f1dbc43d34bdfb6e1b2fbf2a483ca715b088a0e83ffaacc9ffb7822" exitCode=0
Sep 09 20:49:10.565395 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:10.565252    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5" event={"ID":"2caa6df3-5b63-4212-b03c-8909bb2f4f92","Type":"ContainerDied","Data":"3b43e54a0f1dbc43d34bdfb6e1b2fbf2a483ca715b088a0e83ffaacc9ffb7822"}
Sep 09 20:49:10.565395 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:10.565259    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5"
Sep 09 20:49:10.565395 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:10.565322    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5" event={"ID":"2caa6df3-5b63-4212-b03c-8909bb2f4f92","Type":"ContainerDied","Data":"b41cb25330920a375d92885e0a92100fbd307e95e874200b31c51cf6a7d496bb"}
Sep 09 20:49:10.565395 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:10.565345    2556 scope.go:117] "RemoveContainer" containerID="3b43e54a0f1dbc43d34bdfb6e1b2fbf2a483ca715b088a0e83ffaacc9ffb7822"
Sep 09 20:49:10.597224 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:10.597199    2556 scope.go:117] "RemoveContainer" containerID="3b43e54a0f1dbc43d34bdfb6e1b2fbf2a483ca715b088a0e83ffaacc9ffb7822"
Sep 09 20:49:10.597627 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:49:10.597600    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b43e54a0f1dbc43d34bdfb6e1b2fbf2a483ca715b088a0e83ffaacc9ffb7822\": container with ID starting with 3b43e54a0f1dbc43d34bdfb6e1b2fbf2a483ca715b088a0e83ffaacc9ffb7822 not found: ID does not exist" containerID="3b43e54a0f1dbc43d34bdfb6e1b2fbf2a483ca715b088a0e83ffaacc9ffb7822"
Sep 09 20:49:10.597739 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:10.597639    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b43e54a0f1dbc43d34bdfb6e1b2fbf2a483ca715b088a0e83ffaacc9ffb7822"} err="failed to get container status \"3b43e54a0f1dbc43d34bdfb6e1b2fbf2a483ca715b088a0e83ffaacc9ffb7822\": rpc error: code = NotFound desc = could not find container \"3b43e54a0f1dbc43d34bdfb6e1b2fbf2a483ca715b088a0e83ffaacc9ffb7822\": container with ID starting with 3b43e54a0f1dbc43d34bdfb6e1b2fbf2a483ca715b088a0e83ffaacc9ffb7822 not found: ID does not exist"
Sep 09 20:49:10.614562 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:10.614508    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5"]
Sep 09 20:49:10.619281 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:10.619254    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-2beba0-75bcdf65f4-4s6m5"]
Sep 09 20:49:10.896944 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:10.896825    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2caa6df3-5b63-4212-b03c-8909bb2f4f92" path="/var/lib/kubelet/pods/2caa6df3-5b63-4212-b03c-8909bb2f4f92/volumes"
Sep 09 20:49:11.240355 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:11.240319    2556 patch_prober.go:28] interesting pod/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 09 20:49:11.240526 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:11.240374    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r" podUID="d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 09 20:49:11.395406 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:11.395367    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:49:12.245506 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:12.245476    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:49:12.246358 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:12.246342    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:49:14.096617 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:14.096582    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"]
Sep 09 20:49:14.097087 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:14.096822    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r" podUID="d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" containerName="istio-proxy" containerID="cri-o://0997d8e6c0a34fee496f9f25239d511bac84ad5acec869266671168839a4217e" gracePeriod=30
Sep 09 20:49:15.636968 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:15.636935    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"]
Sep 09 20:49:16.392746 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:16.392704    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:49:19.351638 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.351614    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:49:19.375274 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.374589    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-credential-socket\") pod \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") "
Sep 09 20:49:19.375274 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.374666    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-envoy\") pod \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") "
Sep 09 20:49:19.375274 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.374705    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-workload-certs\") pod \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") "
Sep 09 20:49:19.375274 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.374735    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-token\") pod \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") "
Sep 09 20:49:19.375274 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.374776    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-data\") pod \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") "
Sep 09 20:49:19.375274 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.374802    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-workload-socket\") pod \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") "
Sep 09 20:49:19.375274 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.374840    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istiod-ca-cert\") pod \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") "
Sep 09 20:49:19.375274 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.374878    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-podinfo\") pod \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") "
Sep 09 20:49:19.375274 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.374934    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fsb9\" (UniqueName: \"kubernetes.io/projected/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-kube-api-access-8fsb9\") pod \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\" (UID: \"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5\") "
Sep 09 20:49:19.376089 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.375773    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-data" (OuterVolumeSpecName: "istio-data") pod "d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" (UID: "d870d634-d19c-4b9b-b5cf-85ca2eeae4f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:19.376274 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.376247    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" (UID: "d870d634-d19c-4b9b-b5cf-85ca2eeae4f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:19.376387 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.376268    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" (UID: "d870d634-d19c-4b9b-b5cf-85ca2eeae4f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:19.376509 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.376487    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" (UID: "d870d634-d19c-4b9b-b5cf-85ca2eeae4f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:19.376572 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.376511    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" (UID: "d870d634-d19c-4b9b-b5cf-85ca2eeae4f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:49:19.378478 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.378449    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-kube-api-access-8fsb9" (OuterVolumeSpecName: "kube-api-access-8fsb9") pod "d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" (UID: "d870d634-d19c-4b9b-b5cf-85ca2eeae4f5"). InnerVolumeSpecName "kube-api-access-8fsb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:49:19.378969 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.378943    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" (UID: "d870d634-d19c-4b9b-b5cf-85ca2eeae4f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:19.379069 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.378961    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-token" (OuterVolumeSpecName: "istio-token") pod "d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" (UID: "d870d634-d19c-4b9b-b5cf-85ca2eeae4f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:49:19.379564 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.379542    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" (UID: "d870d634-d19c-4b9b-b5cf-85ca2eeae4f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 20:49:19.476397 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.476374    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-podinfo\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:19.476397 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.476396    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fsb9\" (UniqueName: \"kubernetes.io/projected/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-kube-api-access-8fsb9\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:19.476536 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.476406    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-credential-socket\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:19.476536 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.476415    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-envoy\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:19.476536 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.476424    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-workload-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:19.476536 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.476431    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-token\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:19.476536 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.476438    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istio-data\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:19.476536 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.476445    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-workload-socket\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:19.476536 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.476452    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5-istiod-ca-cert\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:19.618383 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.618351    2556 generic.go:356] "Generic (PLEG): container finished" podID="d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" containerID="0997d8e6c0a34fee496f9f25239d511bac84ad5acec869266671168839a4217e" exitCode=0
Sep 09 20:49:19.618481 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.618394    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"
Sep 09 20:49:19.618481 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.618395    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r" event={"ID":"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5","Type":"ContainerDied","Data":"0997d8e6c0a34fee496f9f25239d511bac84ad5acec869266671168839a4217e"}
Sep 09 20:49:19.618585 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.618506    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r" event={"ID":"d870d634-d19c-4b9b-b5cf-85ca2eeae4f5","Type":"ContainerDied","Data":"b98c48d14492f3f59fdfab36a3a2a4ea9df3cac4983e38589512fd352b5db440"}
Sep 09 20:49:19.618585 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.618529    2556 scope.go:117] "RemoveContainer" containerID="0997d8e6c0a34fee496f9f25239d511bac84ad5acec869266671168839a4217e"
Sep 09 20:49:19.630393 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.630371    2556 scope.go:117] "RemoveContainer" containerID="490d7d6d394b3595c51e824c359ce72e0af1aca24f76f3672990e265ff61f4c6"
Sep 09 20:49:19.640233 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.640215    2556 scope.go:117] "RemoveContainer" containerID="0997d8e6c0a34fee496f9f25239d511bac84ad5acec869266671168839a4217e"
Sep 09 20:49:19.640495 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:49:19.640476    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0997d8e6c0a34fee496f9f25239d511bac84ad5acec869266671168839a4217e\": container with ID starting with 0997d8e6c0a34fee496f9f25239d511bac84ad5acec869266671168839a4217e not found: ID does not exist" containerID="0997d8e6c0a34fee496f9f25239d511bac84ad5acec869266671168839a4217e"
Sep 09 20:49:19.640548 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.640503    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0997d8e6c0a34fee496f9f25239d511bac84ad5acec869266671168839a4217e"} err="failed to get container status \"0997d8e6c0a34fee496f9f25239d511bac84ad5acec869266671168839a4217e\": rpc error: code = NotFound desc = could not find container \"0997d8e6c0a34fee496f9f25239d511bac84ad5acec869266671168839a4217e\": container with ID starting with 0997d8e6c0a34fee496f9f25239d511bac84ad5acec869266671168839a4217e not found: ID does not exist"
Sep 09 20:49:19.640548 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.640519    2556 scope.go:117] "RemoveContainer" containerID="490d7d6d394b3595c51e824c359ce72e0af1aca24f76f3672990e265ff61f4c6"
Sep 09 20:49:19.640745 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:49:19.640728    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"490d7d6d394b3595c51e824c359ce72e0af1aca24f76f3672990e265ff61f4c6\": container with ID starting with 490d7d6d394b3595c51e824c359ce72e0af1aca24f76f3672990e265ff61f4c6 not found: ID does not exist" containerID="490d7d6d394b3595c51e824c359ce72e0af1aca24f76f3672990e265ff61f4c6"
Sep 09 20:49:19.640786 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.640749    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490d7d6d394b3595c51e824c359ce72e0af1aca24f76f3672990e265ff61f4c6"} err="failed to get container status \"490d7d6d394b3595c51e824c359ce72e0af1aca24f76f3672990e265ff61f4c6\": rpc error: code = NotFound desc = could not find container \"490d7d6d394b3595c51e824c359ce72e0af1aca24f76f3672990e265ff61f4c6\": container with ID starting with 490d7d6d394b3595c51e824c359ce72e0af1aca24f76f3672990e265ff61f4c6 not found: ID does not exist"
Sep 09 20:49:19.647497 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.647475    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"]
Sep 09 20:49:19.651288 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:19.651268    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-3d3c9036-openshift-default-6fc54fffd4-nl64r"]
Sep 09 20:49:20.892658 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:20.892625    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" path="/var/lib/kubelet/pods/d870d634-d19c-4b9b-b5cf-85ca2eeae4f5/volumes"
Sep 09 20:49:29.298053 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:29.298020    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:49:30.656483 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:30.656440    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh" podUID="60b4490b-6b81-4410-a185-27654a7227dd" containerName="main" containerID="cri-o://be3c71b5f331180d26bb27d32262ebb9686fd9bf626f908af5b7e4ff0ccb3ff8" gracePeriod=45
Sep 09 20:49:30.906092 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:30.906070    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:49:30.978927 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:30.978842    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-model-cache\") pod \"60b4490b-6b81-4410-a185-27654a7227dd\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") "
Sep 09 20:49:30.978927 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:30.978870    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-home\") pod \"60b4490b-6b81-4410-a185-27654a7227dd\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") "
Sep 09 20:49:30.978927 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:30.978921    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-dshm\") pod \"60b4490b-6b81-4410-a185-27654a7227dd\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") "
Sep 09 20:49:30.979180 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:30.978968    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpp2c\" (UniqueName: \"kubernetes.io/projected/60b4490b-6b81-4410-a185-27654a7227dd-kube-api-access-mpp2c\") pod \"60b4490b-6b81-4410-a185-27654a7227dd\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") "
Sep 09 20:49:30.979180 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:30.979026    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b4490b-6b81-4410-a185-27654a7227dd-tls-certs\") pod \"60b4490b-6b81-4410-a185-27654a7227dd\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") "
Sep 09 20:49:30.979180 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:30.979071    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-tmp-dir\") pod \"60b4490b-6b81-4410-a185-27654a7227dd\" (UID: \"60b4490b-6b81-4410-a185-27654a7227dd\") "
Sep 09 20:49:30.979180 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:30.979120    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-home" (OuterVolumeSpecName: "home") pod "60b4490b-6b81-4410-a185-27654a7227dd" (UID: "60b4490b-6b81-4410-a185-27654a7227dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:30.979180 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:30.979144    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "60b4490b-6b81-4410-a185-27654a7227dd" (UID: "60b4490b-6b81-4410-a185-27654a7227dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:30.979420 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:30.979353    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "60b4490b-6b81-4410-a185-27654a7227dd" (UID: "60b4490b-6b81-4410-a185-27654a7227dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:30.979420 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:30.979379    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-model-cache\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:30.979420 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:30.979397    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-home\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:30.981151 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:30.981129    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-dshm" (OuterVolumeSpecName: "dshm") pod "60b4490b-6b81-4410-a185-27654a7227dd" (UID: "60b4490b-6b81-4410-a185-27654a7227dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:30.981300 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:30.981280    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60b4490b-6b81-4410-a185-27654a7227dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60b4490b-6b81-4410-a185-27654a7227dd" (UID: "60b4490b-6b81-4410-a185-27654a7227dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:49:30.981437 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:30.981417    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b4490b-6b81-4410-a185-27654a7227dd-kube-api-access-mpp2c" (OuterVolumeSpecName: "kube-api-access-mpp2c") pod "60b4490b-6b81-4410-a185-27654a7227dd" (UID: "60b4490b-6b81-4410-a185-27654a7227dd"). InnerVolumeSpecName "kube-api-access-mpp2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:49:31.080686 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:31.080659    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b4490b-6b81-4410-a185-27654a7227dd-tls-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:31.080686 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:31.080685    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-tmp-dir\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:31.080853 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:31.080697    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b4490b-6b81-4410-a185-27654a7227dd-dshm\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:31.080853 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:31.080709    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpp2c\" (UniqueName: \"kubernetes.io/projected/60b4490b-6b81-4410-a185-27654a7227dd-kube-api-access-mpp2c\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:31.686678 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:31.686643    2556 generic.go:356] "Generic (PLEG): container finished" podID="60b4490b-6b81-4410-a185-27654a7227dd" containerID="be3c71b5f331180d26bb27d32262ebb9686fd9bf626f908af5b7e4ff0ccb3ff8" exitCode=0
Sep 09 20:49:31.687130 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:31.686689    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"
Sep 09 20:49:31.687130 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:31.686728    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh" event={"ID":"60b4490b-6b81-4410-a185-27654a7227dd","Type":"ContainerDied","Data":"be3c71b5f331180d26bb27d32262ebb9686fd9bf626f908af5b7e4ff0ccb3ff8"}
Sep 09 20:49:31.687130 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:31.686772    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh" event={"ID":"60b4490b-6b81-4410-a185-27654a7227dd","Type":"ContainerDied","Data":"8dc9dc840228c6e21847e16ecc24257d9a7b45bae48674bee952daee8b8e2572"}
Sep 09 20:49:31.687130 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:31.686794    2556 scope.go:117] "RemoveContainer" containerID="be3c71b5f331180d26bb27d32262ebb9686fd9bf626f908af5b7e4ff0ccb3ff8"
Sep 09 20:49:31.696238 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:31.696222    2556 scope.go:117] "RemoveContainer" containerID="be3c71b5f331180d26bb27d32262ebb9686fd9bf626f908af5b7e4ff0ccb3ff8"
Sep 09 20:49:31.696510 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:49:31.696493    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be3c71b5f331180d26bb27d32262ebb9686fd9bf626f908af5b7e4ff0ccb3ff8\": container with ID starting with be3c71b5f331180d26bb27d32262ebb9686fd9bf626f908af5b7e4ff0ccb3ff8 not found: ID does not exist" containerID="be3c71b5f331180d26bb27d32262ebb9686fd9bf626f908af5b7e4ff0ccb3ff8"
Sep 09 20:49:31.696557 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:31.696516    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be3c71b5f331180d26bb27d32262ebb9686fd9bf626f908af5b7e4ff0ccb3ff8"} err="failed to get container status \"be3c71b5f331180d26bb27d32262ebb9686fd9bf626f908af5b7e4ff0ccb3ff8\": rpc error: code = NotFound desc = could not find container \"be3c71b5f331180d26bb27d32262ebb9686fd9bf626f908af5b7e4ff0ccb3ff8\": container with ID starting with be3c71b5f331180d26bb27d32262ebb9686fd9bf626f908af5b7e4ff0ccb3ff8 not found: ID does not exist"
Sep 09 20:49:31.708319 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:31.708288    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"]
Sep 09 20:49:31.711915 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:31.711883    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-2beba0/autores-model-2beba0-kserve-6d6684b655-49cmh"]
Sep 09 20:49:32.893043 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:32.893013    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60b4490b-6b81-4410-a185-27654a7227dd" path="/var/lib/kubelet/pods/60b4490b-6b81-4410-a185-27654a7227dd/volumes"
Sep 09 20:49:45.882924 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:45.882876    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"]
Sep 09 20:49:45.883564 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:45.883517    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2caa6df3-5b63-4212-b03c-8909bb2f4f92" containerName="payload-pre-processing"
Sep 09 20:49:45.883564 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:45.883562    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2caa6df3-5b63-4212-b03c-8909bb2f4f92" containerName="payload-pre-processing"
Sep 09 20:49:45.883767 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:45.883581    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" containerName="istio-proxy"
Sep 09 20:49:45.883767 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:45.883592    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" containerName="istio-proxy"
Sep 09 20:49:45.883767 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:45.883612    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60b4490b-6b81-4410-a185-27654a7227dd" containerName="main"
Sep 09 20:49:45.883767 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:45.883621    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="60b4490b-6b81-4410-a185-27654a7227dd" containerName="main"
Sep 09 20:49:45.883767 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:45.883636    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" containerName="istio-proxy"
Sep 09 20:49:45.883767 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:45.883644    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" containerName="istio-proxy"
Sep 09 20:49:45.883767 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:45.883754    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" containerName="istio-proxy"
Sep 09 20:49:45.883767 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:45.883765    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="60b4490b-6b81-4410-a185-27654a7227dd" containerName="main"
Sep 09 20:49:45.884198 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:45.883774    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2caa6df3-5b63-4212-b03c-8909bb2f4f92" containerName="payload-pre-processing"
Sep 09 20:49:45.888496 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:45.888477    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:45.891573 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:45.891551    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-933e9d\"/\"openshift-service-ca.crt\""
Sep 09 20:49:45.892879 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:45.892860    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-933e9d\"/\"default-dockercfg-85jbr\""
Sep 09 20:49:45.893000 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:45.892860    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-933e9d\"/\"premium-w4-933e9d-kserve-self-signed-certs\""
Sep 09 20:49:45.893000 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:45.892944    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-933e9d\"/\"kube-root-ca.crt\""
Sep 09 20:49:45.902115 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:45.902086    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"]
Sep 09 20:49:46.013451 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:46.013421    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-dshm\") pod \"premium-w4-933e9d-kserve-7795b97f5f-fwqr8\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") " pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:46.013595 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:46.013457    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc18bdc-1c2e-4414-8a04-30dead944112-tls-certs\") pod \"premium-w4-933e9d-kserve-7795b97f5f-fwqr8\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") " pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:46.013595 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:46.013490    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-home\") pod \"premium-w4-933e9d-kserve-7795b97f5f-fwqr8\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") " pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:46.013595 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:46.013549    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-tmp-dir\") pod \"premium-w4-933e9d-kserve-7795b97f5f-fwqr8\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") " pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:46.013708 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:46.013599    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-model-cache\") pod \"premium-w4-933e9d-kserve-7795b97f5f-fwqr8\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") " pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:46.013708 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:46.013663    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkbtk\" (UniqueName: \"kubernetes.io/projected/6bc18bdc-1c2e-4414-8a04-30dead944112-kube-api-access-kkbtk\") pod \"premium-w4-933e9d-kserve-7795b97f5f-fwqr8\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") " pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:46.114040 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:46.114011    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-model-cache\") pod \"premium-w4-933e9d-kserve-7795b97f5f-fwqr8\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") " pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:46.114179 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:46.114066    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkbtk\" (UniqueName: \"kubernetes.io/projected/6bc18bdc-1c2e-4414-8a04-30dead944112-kube-api-access-kkbtk\") pod \"premium-w4-933e9d-kserve-7795b97f5f-fwqr8\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") " pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:46.114179 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:46.114125    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-dshm\") pod \"premium-w4-933e9d-kserve-7795b97f5f-fwqr8\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") " pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:46.114179 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:46.114155    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc18bdc-1c2e-4414-8a04-30dead944112-tls-certs\") pod \"premium-w4-933e9d-kserve-7795b97f5f-fwqr8\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") " pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:46.114331 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:46.114194    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-home\") pod \"premium-w4-933e9d-kserve-7795b97f5f-fwqr8\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") " pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:46.114331 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:46.114220    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-tmp-dir\") pod \"premium-w4-933e9d-kserve-7795b97f5f-fwqr8\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") " pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:46.114480 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:46.114460    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-model-cache\") pod \"premium-w4-933e9d-kserve-7795b97f5f-fwqr8\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") " pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:46.114563 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:46.114517    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-tmp-dir\") pod \"premium-w4-933e9d-kserve-7795b97f5f-fwqr8\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") " pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:46.114618 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:46.114578    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-home\") pod \"premium-w4-933e9d-kserve-7795b97f5f-fwqr8\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") " pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:46.116349 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:46.116327    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-dshm\") pod \"premium-w4-933e9d-kserve-7795b97f5f-fwqr8\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") " pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:46.116639 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:46.116621    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc18bdc-1c2e-4414-8a04-30dead944112-tls-certs\") pod \"premium-w4-933e9d-kserve-7795b97f5f-fwqr8\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") " pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:46.123494 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:46.123472    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkbtk\" (UniqueName: \"kubernetes.io/projected/6bc18bdc-1c2e-4414-8a04-30dead944112-kube-api-access-kkbtk\") pod \"premium-w4-933e9d-kserve-7795b97f5f-fwqr8\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") " pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:46.197863 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:46.197785    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:46.468694 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:46.468671    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"]
Sep 09 20:49:46.470456 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:49:46.470424    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bc18bdc_1c2e_4414_8a04_30dead944112.slice/crio-06ce7edd694dd793b12af431d15d7b5482f127360d1983a2bb40339501a41527 WatchSource:0}: Error finding container 06ce7edd694dd793b12af431d15d7b5482f127360d1983a2bb40339501a41527: Status 404 returned error can't find the container with id 06ce7edd694dd793b12af431d15d7b5482f127360d1983a2bb40339501a41527
Sep 09 20:49:46.779965 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:46.779935    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8" event={"ID":"6bc18bdc-1c2e-4414-8a04-30dead944112","Type":"ContainerStarted","Data":"2ee2f7841d59af77792edaf2ec2fbdc6aa461167ac72c9296a2b45bb4142584b"}
Sep 09 20:49:46.780134 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:46.779973    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8" event={"ID":"6bc18bdc-1c2e-4414-8a04-30dead944112","Type":"ContainerStarted","Data":"06ce7edd694dd793b12af431d15d7b5482f127360d1983a2bb40339501a41527"}
Sep 09 20:49:46.798927 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:46.798860    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8" podStartSLOduration=1.7988470109999999 podStartE2EDuration="1.798847011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:46.797177884 +0000 UTC m=+1032.477138200" watchObservedRunningTime="2026-09-09 20:49:46.798847011 +0000 UTC m=+1032.478807319"
Sep 09 20:49:47.202043 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:47.201963    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:49:52.715888 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:52.715857    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"]
Sep 09 20:49:52.716320 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:52.716094    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb" podUID="4eb9f926-a0c6-40df-b3af-7bba52226164" containerName="istio-proxy" containerID="cri-o://e9f4b1939bc7bdeaf7a8842b217cba35377b93fa0be9eeea6676b3b120eb7a24" gracePeriod=30
Sep 09 20:49:53.429929 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.429886    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"]
Sep 09 20:49:53.430402 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.430384    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d870d634-d19c-4b9b-b5cf-85ca2eeae4f5" containerName="istio-proxy"
Sep 09 20:49:53.433697 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.433682    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.436258 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.436233    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-2d6f6c-openshift-default-dockercfg-cjdbz\""
Sep 09 20:49:53.446169 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.446144    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"]
Sep 09 20:49:53.583318 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.583285    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-workload-socket\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.583318 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.583320    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-envoy\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.583522 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.583342    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-token\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.583522 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.583366    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-credential-socket\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.583522 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.583382    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-workload-certs\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.583522 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.583423    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-data\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.583522 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.583458    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz2bx\" (UniqueName: \"kubernetes.io/projected/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-kube-api-access-pz2bx\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.583522 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.583479    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istiod-ca-cert\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.583522 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.583503    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-podinfo\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.684408 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.684325    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-workload-socket\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.684408 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.684358    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-envoy\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.684408 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.684383    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-token\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.684641 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.684512    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-credential-socket\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.684641 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.684540    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-workload-certs\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.684641 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.684619    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-data\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.684795 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.684687    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz2bx\" (UniqueName: \"kubernetes.io/projected/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-kube-api-access-pz2bx\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.684795 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.684754    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istiod-ca-cert\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.684920 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.684801    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-podinfo\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.684986 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.684929    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-credential-socket\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.685042 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.684992    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-data\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.685104 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.685033    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-workload-certs\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.685162 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.685131    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-workload-socket\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.685547 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.685526    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istiod-ca-cert\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.687064 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.687041    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-envoy\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.687362 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.687342    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-podinfo\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.697773 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.697746    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz2bx\" (UniqueName: \"kubernetes.io/projected/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-kube-api-access-pz2bx\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.697859 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.697754    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-token\") pod \"e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") " pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:53.744606 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:53.744586    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:54.021855 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:54.021830    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"]
Sep 09 20:49:54.023638 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:49:54.023615    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0a3043_56fc_4f66_bd1b_fca0fb23eef1.slice/crio-f29306e5e1cbece4637bfb679fad32322f0334ebc8d8d083f62f17353535a6f0 WatchSource:0}: Error finding container f29306e5e1cbece4637bfb679fad32322f0334ebc8d8d083f62f17353535a6f0: Status 404 returned error can't find the container with id f29306e5e1cbece4637bfb679fad32322f0334ebc8d8d083f62f17353535a6f0
Sep 09 20:49:54.025730 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:54.025702    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:49:54.025792 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:54.025760    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:49:54.025792 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:54.025789    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:49:54.827281 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:54.827247    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8" event={"ID":"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1","Type":"ContainerStarted","Data":"362909a3f9a539c039ae11c66046f8fba735bd63f1e9494ff7852fb8740caecd"}
Sep 09 20:49:54.827694 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:54.827302    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8" event={"ID":"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1","Type":"ContainerStarted","Data":"f29306e5e1cbece4637bfb679fad32322f0334ebc8d8d083f62f17353535a6f0"}
Sep 09 20:49:54.863372 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:54.863317    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8" podStartSLOduration=1.8633019769999999 podStartE2EDuration="1.863301977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:54.861542313 +0000 UTC m=+1040.541502621" watchObservedRunningTime="2026-09-09 20:49:54.863301977 +0000 UTC m=+1040.543262286"
Sep 09 20:49:55.744795 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:55.744763    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:55.749800 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:55.749776    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:55.832352 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:55.832320    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:55.833129 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:55.833110    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:49:56.198857 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:56.198776    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:56.198857 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:56.198817    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:56.211372 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:56.211351    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:56.850297 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:56.850272    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 20:49:57.846851 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:57.846825    2556 generic.go:356] "Generic (PLEG): container finished" podID="4eb9f926-a0c6-40df-b3af-7bba52226164" containerID="e9f4b1939bc7bdeaf7a8842b217cba35377b93fa0be9eeea6676b3b120eb7a24" exitCode=0
Sep 09 20:49:57.846999 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:57.846894    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb" event={"ID":"4eb9f926-a0c6-40df-b3af-7bba52226164","Type":"ContainerDied","Data":"e9f4b1939bc7bdeaf7a8842b217cba35377b93fa0be9eeea6676b3b120eb7a24"}
Sep 09 20:49:57.846999 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:57.846965    2556 scope.go:117] "RemoveContainer" containerID="cdb6d3122b8de657a4e9efb1ec76e8593645f29c7e96b6ad22b6ae32a43efa4e"
Sep 09 20:49:57.989105 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:57.989081    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:49:58.020977 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.020950    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkcv4\" (UniqueName: \"kubernetes.io/projected/4eb9f926-a0c6-40df-b3af-7bba52226164-kube-api-access-rkcv4\") pod \"4eb9f926-a0c6-40df-b3af-7bba52226164\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") "
Sep 09 20:49:58.021123 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.021020    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-workload-certs\") pod \"4eb9f926-a0c6-40df-b3af-7bba52226164\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") "
Sep 09 20:49:58.021123 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.021099    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-credential-socket\") pod \"4eb9f926-a0c6-40df-b3af-7bba52226164\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") "
Sep 09 20:49:58.021255 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.021199    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-token\") pod \"4eb9f926-a0c6-40df-b3af-7bba52226164\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") "
Sep 09 20:49:58.021388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.021357    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4eb9f926-a0c6-40df-b3af-7bba52226164" (UID: "4eb9f926-a0c6-40df-b3af-7bba52226164"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:58.021388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.021371    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4eb9f926-a0c6-40df-b3af-7bba52226164" (UID: "4eb9f926-a0c6-40df-b3af-7bba52226164"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:58.021518 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.021501    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb9f926-a0c6-40df-b3af-7bba52226164-istiod-ca-cert\") pod \"4eb9f926-a0c6-40df-b3af-7bba52226164\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") "
Sep 09 20:49:58.021677 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.021570    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-data\") pod \"4eb9f926-a0c6-40df-b3af-7bba52226164\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") "
Sep 09 20:49:58.021864 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.021840    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-envoy\") pod \"4eb9f926-a0c6-40df-b3af-7bba52226164\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") "
Sep 09 20:49:58.021864 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.021772    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb9f926-a0c6-40df-b3af-7bba52226164-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4eb9f926-a0c6-40df-b3af-7bba52226164" (UID: "4eb9f926-a0c6-40df-b3af-7bba52226164"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:49:58.022073 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.021788    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-data" (OuterVolumeSpecName: "istio-data") pod "4eb9f926-a0c6-40df-b3af-7bba52226164" (UID: "4eb9f926-a0c6-40df-b3af-7bba52226164"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:58.022073 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.021891    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-workload-socket\") pod \"4eb9f926-a0c6-40df-b3af-7bba52226164\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") "
Sep 09 20:49:58.022073 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.021957    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-podinfo\") pod \"4eb9f926-a0c6-40df-b3af-7bba52226164\" (UID: \"4eb9f926-a0c6-40df-b3af-7bba52226164\") "
Sep 09 20:49:58.022345 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.022168    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4eb9f926-a0c6-40df-b3af-7bba52226164" (UID: "4eb9f926-a0c6-40df-b3af-7bba52226164"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:58.022602 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.022586    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb9f926-a0c6-40df-b3af-7bba52226164-istiod-ca-cert\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:58.022720 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.022709    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-data\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:58.022799 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.022789    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-workload-socket\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:58.022888 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.022867    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-workload-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:58.022990 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.022980    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-credential-socket\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:58.023739 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.023714    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb9f926-a0c6-40df-b3af-7bba52226164-kube-api-access-rkcv4" (OuterVolumeSpecName: "kube-api-access-rkcv4") pod "4eb9f926-a0c6-40df-b3af-7bba52226164" (UID: "4eb9f926-a0c6-40df-b3af-7bba52226164"). InnerVolumeSpecName "kube-api-access-rkcv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:49:58.023824 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.023719    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-token" (OuterVolumeSpecName: "istio-token") pod "4eb9f926-a0c6-40df-b3af-7bba52226164" (UID: "4eb9f926-a0c6-40df-b3af-7bba52226164"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:49:58.024350 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.024328    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4eb9f926-a0c6-40df-b3af-7bba52226164" (UID: "4eb9f926-a0c6-40df-b3af-7bba52226164"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:58.024650 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.024626    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4eb9f926-a0c6-40df-b3af-7bba52226164" (UID: "4eb9f926-a0c6-40df-b3af-7bba52226164"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 20:49:58.124320 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.124291    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-envoy\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:58.124320 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.124315    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-podinfo\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:58.124320 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.124325    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkcv4\" (UniqueName: \"kubernetes.io/projected/4eb9f926-a0c6-40df-b3af-7bba52226164-kube-api-access-rkcv4\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:58.124529 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.124334    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb9f926-a0c6-40df-b3af-7bba52226164-istio-token\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:49:58.853133 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.853103    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"
Sep 09 20:49:58.853302 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.853156    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb" event={"ID":"4eb9f926-a0c6-40df-b3af-7bba52226164","Type":"ContainerDied","Data":"beeb0ccc0ea1227f5cd5abcb722421b4e01a3693b03464bf687b1bc14235cb8b"}
Sep 09 20:49:58.853302 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.853210    2556 scope.go:117] "RemoveContainer" containerID="e9f4b1939bc7bdeaf7a8842b217cba35377b93fa0be9eeea6676b3b120eb7a24"
Sep 09 20:49:58.880249 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.880225    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg"]
Sep 09 20:49:58.880655 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.880641    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eb9f926-a0c6-40df-b3af-7bba52226164" containerName="istio-proxy"
Sep 09 20:49:58.880701 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.880658    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eb9f926-a0c6-40df-b3af-7bba52226164" containerName="istio-proxy"
Sep 09 20:49:58.880701 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.880669    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eb9f926-a0c6-40df-b3af-7bba52226164" containerName="istio-proxy"
Sep 09 20:49:58.880701 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.880674    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eb9f926-a0c6-40df-b3af-7bba52226164" containerName="istio-proxy"
Sep 09 20:49:58.880794 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.880754    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eb9f926-a0c6-40df-b3af-7bba52226164" containerName="istio-proxy"
Sep 09 20:49:58.885057 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.885041    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg"
Sep 09 20:49:58.888309 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.888283    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-2d6f6c-dockercfg-ppp5m\""
Sep 09 20:49:58.889392 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.889374    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-2d6f6c\""
Sep 09 20:49:58.893107 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.893090    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"]
Sep 09 20:49:58.904764 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.904722    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg"]
Sep 09 20:49:58.907057 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.907036    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-2beba0-openshift-default-696968fc9d-d68zb"]
Sep 09 20:49:58.932777 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.932750    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92075b8b-5763-446a-ae3c-7f2ddc4124b7-plugins-config-volume\") pod \"payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg\" (UID: \"92075b8b-5763-446a-ae3c-7f2ddc4124b7\") " pod="openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg"
Sep 09 20:49:58.932875 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:58.932797    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jkjh\" (UniqueName: \"kubernetes.io/projected/92075b8b-5763-446a-ae3c-7f2ddc4124b7-kube-api-access-9jkjh\") pod \"payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg\" (UID: \"92075b8b-5763-446a-ae3c-7f2ddc4124b7\") " pod="openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg"
Sep 09 20:49:59.034237 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:59.034212    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92075b8b-5763-446a-ae3c-7f2ddc4124b7-plugins-config-volume\") pod \"payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg\" (UID: \"92075b8b-5763-446a-ae3c-7f2ddc4124b7\") " pod="openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg"
Sep 09 20:49:59.034527 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:59.034266    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jkjh\" (UniqueName: \"kubernetes.io/projected/92075b8b-5763-446a-ae3c-7f2ddc4124b7-kube-api-access-9jkjh\") pod \"payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg\" (UID: \"92075b8b-5763-446a-ae3c-7f2ddc4124b7\") " pod="openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg"
Sep 09 20:49:59.034885 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:59.034866    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92075b8b-5763-446a-ae3c-7f2ddc4124b7-plugins-config-volume\") pod \"payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg\" (UID: \"92075b8b-5763-446a-ae3c-7f2ddc4124b7\") " pod="openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg"
Sep 09 20:49:59.046586 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:59.046564    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jkjh\" (UniqueName: \"kubernetes.io/projected/92075b8b-5763-446a-ae3c-7f2ddc4124b7-kube-api-access-9jkjh\") pod \"payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg\" (UID: \"92075b8b-5763-446a-ae3c-7f2ddc4124b7\") " pod="openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg"
Sep 09 20:49:59.195517 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:59.195459    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg"
Sep 09 20:49:59.470815 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:59.470793    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg"]
Sep 09 20:49:59.472749 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:49:59.472722    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92075b8b_5763_446a_ae3c_7f2ddc4124b7.slice/crio-d217aa6e96101da95d01417de40991ef1cc6df8725c341dac4a1850fb8cc4388 WatchSource:0}: Error finding container d217aa6e96101da95d01417de40991ef1cc6df8725c341dac4a1850fb8cc4388: Status 404 returned error can't find the container with id d217aa6e96101da95d01417de40991ef1cc6df8725c341dac4a1850fb8cc4388
Sep 09 20:49:59.862422 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:59.862385    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg" event={"ID":"92075b8b-5763-446a-ae3c-7f2ddc4124b7","Type":"ContainerStarted","Data":"7b7b8b90a97075bc1bb972716987bd41cb56c7e22a5815858639e90234d4e47d"}
Sep 09 20:49:59.862615 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:59.862436    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg" event={"ID":"92075b8b-5763-446a-ae3c-7f2ddc4124b7","Type":"ContainerStarted","Data":"d217aa6e96101da95d01417de40991ef1cc6df8725c341dac4a1850fb8cc4388"}
Sep 09 20:49:59.862615 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:59.862479    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg"
Sep 09 20:49:59.883226 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:49:59.883176    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg" podStartSLOduration=1.883162719 podStartE2EDuration="1.883162719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:59.881178142 +0000 UTC m=+1045.561138453" watchObservedRunningTime="2026-09-09 20:49:59.883162719 +0000 UTC m=+1045.563123082"
Sep 09 20:50:00.893055 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:00.893022    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eb9f926-a0c6-40df-b3af-7bba52226164" path="/var/lib/kubelet/pods/4eb9f926-a0c6-40df-b3af-7bba52226164/volumes"
Sep 09 20:50:04.597560 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:04.597526    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:50:10.871133 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:10.871100    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg"
Sep 09 20:50:16.960433 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:16.960399    2556 generic.go:356] "Generic (PLEG): container finished" podID="fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" containerID="362909a3f9a539c039ae11c66046f8fba735bd63f1e9494ff7852fb8740caecd" exitCode=0
Sep 09 20:50:16.960849 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:16.960478    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8" event={"ID":"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1","Type":"ContainerDied","Data":"362909a3f9a539c039ae11c66046f8fba735bd63f1e9494ff7852fb8740caecd"}
Sep 09 20:50:16.960943 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:16.960927    2556 scope.go:117] "RemoveContainer" containerID="362909a3f9a539c039ae11c66046f8fba735bd63f1e9494ff7852fb8740caecd"
Sep 09 20:50:16.962104 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:16.962068    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:50:16.962173 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:16.962152    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:50:16.962219 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:16.962203    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:50:17.967142 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:17.967109    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8" event={"ID":"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1","Type":"ContainerStarted","Data":"637ef1e4644cfefb3dc6b6c5400d66dff8367d8a3275a126898148cbc68944a4"}
Sep 09 20:50:18.745730 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:18.745691    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:50:18.745976 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:18.745949    2556 patch_prober.go:28] interesting pod/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 09 20:50:18.746076 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:18.746027    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8" podUID="fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 09 20:50:19.745363 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:19.745325    2556 patch_prober.go:28] interesting pod/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 09 20:50:19.745775 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:19.745380    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8" podUID="fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 09 20:50:20.745124 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:20.745089    2556 patch_prober.go:28] interesting pod/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 09 20:50:20.745296 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:20.745149    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8" podUID="fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 09 20:50:21.745294 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:21.745256    2556 patch_prober.go:28] interesting pod/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 09 20:50:21.745679 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:21.745311    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8" podUID="fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 09 20:50:22.745891 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:22.745850    2556 patch_prober.go:28] interesting pod/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 09 20:50:22.746374 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:22.745948    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8" podUID="fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 09 20:50:23.402517 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:23.402481    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:50:23.749230 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:23.749206    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:50:23.749622 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:23.749482    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:50:23.750026 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:23.750008    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:50:41.798055 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:41.798022    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:50:49.697295 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.697258    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"]
Sep 09 20:50:49.697837 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.697814    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eb9f926-a0c6-40df-b3af-7bba52226164" containerName="istio-proxy"
Sep 09 20:50:49.702578 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.702559    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.705363 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.705343    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-999c8b38-openshift-default-dockercfg-xws2n\""
Sep 09 20:50:49.715595 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.715573    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"]
Sep 09 20:50:49.799004 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.798971    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h44p\" (UniqueName: \"kubernetes.io/projected/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-kube-api-access-4h44p\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.799173 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.799018    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istiod-ca-cert\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.799173 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.799061    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-podinfo\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.799173 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.799081    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-token\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.799419 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.799207    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-credential-socket\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.799419 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.799249    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-workload-socket\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.799419 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.799312    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-workload-certs\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.799419 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.799343    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-envoy\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.799419 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.799368    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-data\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.899823 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.899794    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-credential-socket\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.899823 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.899827    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-workload-socket\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.900041 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.899856    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-workload-certs\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.900041 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.899872    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-envoy\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.900041 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.899947    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-data\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.900041 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.900011    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h44p\" (UniqueName: \"kubernetes.io/projected/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-kube-api-access-4h44p\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.900256 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.900047    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istiod-ca-cert\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.900256 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.900089    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-podinfo\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.900256 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.900115    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-token\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.900256 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.900233    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-workload-socket\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.900256 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.900255    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-credential-socket\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.900522 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.900309    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-workload-certs\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.900734 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.900695    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-data\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.900849 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.900782    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istiod-ca-cert\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.902405 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.902378    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-envoy\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.902495 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.902444    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-podinfo\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.910068 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.910040    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-token\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:49.910169 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:49.910150    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h44p\" (UniqueName: \"kubernetes.io/projected/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-kube-api-access-4h44p\") pod \"e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") " pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:50.014804 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:50.014777    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:50.329084 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:50:50.329056    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c32eaef_e67e_4fef_a1f1_a22abe2dc45b.slice/crio-7c037cd75db09b6dc2febc0ef538c45e976aeee76b678fa3936d162c5a0ae668 WatchSource:0}: Error finding container 7c037cd75db09b6dc2febc0ef538c45e976aeee76b678fa3936d162c5a0ae668: Status 404 returned error can't find the container with id 7c037cd75db09b6dc2febc0ef538c45e976aeee76b678fa3936d162c5a0ae668
Sep 09 20:50:50.331376 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:50.331344    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:50:50.331482 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:50.331410    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:50:50.331482 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:50.331438    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:50:50.333364 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:50.333346    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"]
Sep 09 20:50:51.151706 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:51.151667    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78" event={"ID":"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b","Type":"ContainerStarted","Data":"5df5183d84ceb81bf0d4376d904f9be9690cc746dce5c397000d67f2d668b404"}
Sep 09 20:50:51.152171 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:51.151720    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78" event={"ID":"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b","Type":"ContainerStarted","Data":"7c037cd75db09b6dc2febc0ef538c45e976aeee76b678fa3936d162c5a0ae668"}
Sep 09 20:50:51.174843 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:51.174801    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78" podStartSLOduration=2.174787151 podStartE2EDuration="2.174787151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:51.172570163 +0000 UTC m=+1096.852530473" watchObservedRunningTime="2026-09-09 20:50:51.174787151 +0000 UTC m=+1096.854747459"
Sep 09 20:50:52.015965 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:52.015921    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:52.020826 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:52.020799    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:52.157413 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:52.157383    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:50:52.158157 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:50:52.158140    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:51:03.299981 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:03.299946    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:51:14.279818 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:14.279780    2556 generic.go:356] "Generic (PLEG): container finished" podID="7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" containerID="5df5183d84ceb81bf0d4376d904f9be9690cc746dce5c397000d67f2d668b404" exitCode=0
Sep 09 20:51:14.280302 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:14.279852    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78" event={"ID":"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b","Type":"ContainerDied","Data":"5df5183d84ceb81bf0d4376d904f9be9690cc746dce5c397000d67f2d668b404"}
Sep 09 20:51:14.280302 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:14.280246    2556 scope.go:117] "RemoveContainer" containerID="5df5183d84ceb81bf0d4376d904f9be9690cc746dce5c397000d67f2d668b404"
Sep 09 20:51:14.281362 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:14.281333    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:51:14.281458 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:14.281411    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:51:14.281520 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:14.281461    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:51:14.901543 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:14.901512    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"]
Sep 09 20:51:14.905797 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:14.905771    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:14.907668 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:14.907637    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"]
Sep 09 20:51:14.908749 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:14.908728    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b2dd30\"/\"kube-root-ca.crt\""
Sep 09 20:51:14.910293 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:14.910272    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b2dd30\"/\"openshift-service-ca.crt\""
Sep 09 20:51:14.910398 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:14.910286    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b2dd30\"/\"test-model-b2dd30-kserve-self-signed-certs\""
Sep 09 20:51:14.910808 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:14.910786    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b2dd30\"/\"default-dockercfg-7b4pn\""
Sep 09 20:51:15.040554 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:15.040522    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-dshm\") pod \"test-model-b2dd30-kserve-f7b7db955-cgw6q\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") " pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:15.040689 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:15.040581    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcc42\" (UniqueName: \"kubernetes.io/projected/65e66050-c303-4cfa-88ad-6bad1b65eb0b-kube-api-access-mcc42\") pod \"test-model-b2dd30-kserve-f7b7db955-cgw6q\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") " pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:15.040689 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:15.040633    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-tmp-dir\") pod \"test-model-b2dd30-kserve-f7b7db955-cgw6q\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") " pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:15.040689 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:15.040656    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-home\") pod \"test-model-b2dd30-kserve-f7b7db955-cgw6q\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") " pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:15.040801 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:15.040775    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-model-cache\") pod \"test-model-b2dd30-kserve-f7b7db955-cgw6q\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") " pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:15.040847 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:15.040836    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65e66050-c303-4cfa-88ad-6bad1b65eb0b-tls-certs\") pod \"test-model-b2dd30-kserve-f7b7db955-cgw6q\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") " pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:15.141508 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:15.141479    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-model-cache\") pod \"test-model-b2dd30-kserve-f7b7db955-cgw6q\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") " pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:15.141678 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:15.141562    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65e66050-c303-4cfa-88ad-6bad1b65eb0b-tls-certs\") pod \"test-model-b2dd30-kserve-f7b7db955-cgw6q\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") " pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:15.141678 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:15.141604    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-dshm\") pod \"test-model-b2dd30-kserve-f7b7db955-cgw6q\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") " pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:15.141678 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:15.141662    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcc42\" (UniqueName: \"kubernetes.io/projected/65e66050-c303-4cfa-88ad-6bad1b65eb0b-kube-api-access-mcc42\") pod \"test-model-b2dd30-kserve-f7b7db955-cgw6q\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") " pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:15.141853 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:15.141695    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-tmp-dir\") pod \"test-model-b2dd30-kserve-f7b7db955-cgw6q\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") " pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:15.141853 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:15.141777    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-home\") pod \"test-model-b2dd30-kserve-f7b7db955-cgw6q\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") " pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:15.141982 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:15.141852    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-model-cache\") pod \"test-model-b2dd30-kserve-f7b7db955-cgw6q\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") " pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:15.142218 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:15.142192    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-tmp-dir\") pod \"test-model-b2dd30-kserve-f7b7db955-cgw6q\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") " pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:15.142353 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:15.142222    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-home\") pod \"test-model-b2dd30-kserve-f7b7db955-cgw6q\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") " pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:15.144229 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:15.144200    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-dshm\") pod \"test-model-b2dd30-kserve-f7b7db955-cgw6q\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") " pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:15.144555 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:15.144537    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65e66050-c303-4cfa-88ad-6bad1b65eb0b-tls-certs\") pod \"test-model-b2dd30-kserve-f7b7db955-cgw6q\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") " pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:15.151320 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:15.151300    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcc42\" (UniqueName: \"kubernetes.io/projected/65e66050-c303-4cfa-88ad-6bad1b65eb0b-kube-api-access-mcc42\") pod \"test-model-b2dd30-kserve-f7b7db955-cgw6q\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") " pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:15.216588 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:15.216564    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:15.288315 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:15.288287    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78" event={"ID":"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b","Type":"ContainerStarted","Data":"b29780b2f22951bbe4e0b825f251826442b057877d64db25a1ff393fcdeaaf6b"}
Sep 09 20:51:15.508670 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:15.508644    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"]
Sep 09 20:51:15.510979 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:51:15.510947    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65e66050_c303_4cfa_88ad_6bad1b65eb0b.slice/crio-ddb8045fc61e1d010cf7cc673b6863995792ed3f7e612cbdc1a363ef1cfdd60f WatchSource:0}: Error finding container ddb8045fc61e1d010cf7cc673b6863995792ed3f7e612cbdc1a363ef1cfdd60f: Status 404 returned error can't find the container with id ddb8045fc61e1d010cf7cc673b6863995792ed3f7e612cbdc1a363ef1cfdd60f
Sep 09 20:51:16.015332 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:16.015299    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:51:16.015501 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:16.015475    2556 patch_prober.go:28] interesting pod/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 09 20:51:16.015546 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:16.015524    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78" podUID="7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 09 20:51:16.298424 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:16.298328    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q" event={"ID":"65e66050-c303-4cfa-88ad-6bad1b65eb0b","Type":"ContainerStarted","Data":"be2fe42661b7e52d9792767d1eeaa264771cb4a11bbb1649996146abd2a0eafb"}
Sep 09 20:51:16.298424 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:16.298374    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q" event={"ID":"65e66050-c303-4cfa-88ad-6bad1b65eb0b","Type":"ContainerStarted","Data":"ddb8045fc61e1d010cf7cc673b6863995792ed3f7e612cbdc1a363ef1cfdd60f"}
Sep 09 20:51:16.317645 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:16.317591    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q" podStartSLOduration=2.317575093 podStartE2EDuration="2.317575093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:51:16.315934262 +0000 UTC m=+1121.995894572" watchObservedRunningTime="2026-09-09 20:51:16.317575093 +0000 UTC m=+1121.997535440"
Sep 09 20:51:17.016105 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:17.016072    2556 patch_prober.go:28] interesting pod/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 09 20:51:17.016265 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:17.016124    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78" podUID="7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 09 20:51:18.016038 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:18.016001    2556 patch_prober.go:28] interesting pod/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 09 20:51:18.016417 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:18.016053    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78" podUID="7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 09 20:51:19.015248 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:19.015210    2556 patch_prober.go:28] interesting pod/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 09 20:51:19.015420 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:19.015268    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78" podUID="7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 09 20:51:20.016129 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:20.016094    2556 patch_prober.go:28] interesting pod/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 09 20:51:20.016531 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:20.016152    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78" podUID="7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 09 20:51:21.102517 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:21.102484    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:51:21.102985 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:21.102800    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:51:21.103723 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:21.103705    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:51:25.216863 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:25.216827    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:25.216863 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:25.216868    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:25.228980 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:25.228957    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:25.365678 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:25.365654    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:51:26.299315 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:26.299282    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:51:45.142002 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:45.141971    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"]
Sep 09 20:51:45.142629 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:45.142242    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78" podUID="7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" containerName="istio-proxy" containerID="cri-o://b29780b2f22951bbe4e0b825f251826442b057877d64db25a1ff393fcdeaaf6b" gracePeriod=30
Sep 09 20:51:48.501429 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:48.501399    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:51:50.390638 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.390614    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:51:50.477522 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.477435    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-workload-certs\") pod \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") "
Sep 09 20:51:50.477672 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.477519    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-workload-socket\") pod \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") "
Sep 09 20:51:50.477672 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.477565    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-podinfo\") pod \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") "
Sep 09 20:51:50.477672 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.477593    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-envoy\") pod \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") "
Sep 09 20:51:50.477672 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.477629    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-token\") pod \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") "
Sep 09 20:51:50.477672 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.477655    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-data\") pod \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") "
Sep 09 20:51:50.477976 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.477689    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istiod-ca-cert\") pod \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") "
Sep 09 20:51:50.477976 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.477731    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4h44p\" (UniqueName: \"kubernetes.io/projected/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-kube-api-access-4h44p\") pod \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") "
Sep 09 20:51:50.477976 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.477776    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-credential-socket\") pod \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\" (UID: \"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b\") "
Sep 09 20:51:50.477976 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.477778    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" (UID: "7c32eaef-e67e-4fef-a1f1-a22abe2dc45b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:51:50.478194 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.477970    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" (UID: "7c32eaef-e67e-4fef-a1f1-a22abe2dc45b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:51:50.478194 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.478127    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" (UID: "7c32eaef-e67e-4fef-a1f1-a22abe2dc45b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:51:50.478194 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.478142    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" (UID: "7c32eaef-e67e-4fef-a1f1-a22abe2dc45b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:51:50.478355 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.478267    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-workload-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:51:50.478355 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.478313    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-workload-socket\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:51:50.478355 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.478329    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-data\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:51:50.478355 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.478343    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-credential-socket\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:51:50.478834 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.478452    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" (UID: "7c32eaef-e67e-4fef-a1f1-a22abe2dc45b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:51:50.480161 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.480110    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" (UID: "7c32eaef-e67e-4fef-a1f1-a22abe2dc45b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 20:51:50.480293 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.480274    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-kube-api-access-4h44p" (OuterVolumeSpecName: "kube-api-access-4h44p") pod "7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" (UID: "7c32eaef-e67e-4fef-a1f1-a22abe2dc45b"). InnerVolumeSpecName "kube-api-access-4h44p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:51:50.480470 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.480422    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" (UID: "7c32eaef-e67e-4fef-a1f1-a22abe2dc45b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:51:50.480470 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.480432    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" (UID: "7c32eaef-e67e-4fef-a1f1-a22abe2dc45b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:51:50.497113 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.497091    2556 generic.go:356] "Generic (PLEG): container finished" podID="7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" containerID="b29780b2f22951bbe4e0b825f251826442b057877d64db25a1ff393fcdeaaf6b" exitCode=0
Sep 09 20:51:50.497199 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.497136    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"
Sep 09 20:51:50.497199 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.497165    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78" event={"ID":"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b","Type":"ContainerDied","Data":"b29780b2f22951bbe4e0b825f251826442b057877d64db25a1ff393fcdeaaf6b"}
Sep 09 20:51:50.497275 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.497234    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78" event={"ID":"7c32eaef-e67e-4fef-a1f1-a22abe2dc45b","Type":"ContainerDied","Data":"7c037cd75db09b6dc2febc0ef538c45e976aeee76b678fa3936d162c5a0ae668"}
Sep 09 20:51:50.497275 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.497257    2556 scope.go:117] "RemoveContainer" containerID="b29780b2f22951bbe4e0b825f251826442b057877d64db25a1ff393fcdeaaf6b"
Sep 09 20:51:50.506226 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.506210    2556 scope.go:117] "RemoveContainer" containerID="5df5183d84ceb81bf0d4376d904f9be9690cc746dce5c397000d67f2d668b404"
Sep 09 20:51:50.514232 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.514217    2556 scope.go:117] "RemoveContainer" containerID="b29780b2f22951bbe4e0b825f251826442b057877d64db25a1ff393fcdeaaf6b"
Sep 09 20:51:50.514503 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:51:50.514486    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b29780b2f22951bbe4e0b825f251826442b057877d64db25a1ff393fcdeaaf6b\": container with ID starting with b29780b2f22951bbe4e0b825f251826442b057877d64db25a1ff393fcdeaaf6b not found: ID does not exist" containerID="b29780b2f22951bbe4e0b825f251826442b057877d64db25a1ff393fcdeaaf6b"
Sep 09 20:51:50.514550 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.514510    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b29780b2f22951bbe4e0b825f251826442b057877d64db25a1ff393fcdeaaf6b"} err="failed to get container status \"b29780b2f22951bbe4e0b825f251826442b057877d64db25a1ff393fcdeaaf6b\": rpc error: code = NotFound desc = could not find container \"b29780b2f22951bbe4e0b825f251826442b057877d64db25a1ff393fcdeaaf6b\": container with ID starting with b29780b2f22951bbe4e0b825f251826442b057877d64db25a1ff393fcdeaaf6b not found: ID does not exist"
Sep 09 20:51:50.514550 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.514527    2556 scope.go:117] "RemoveContainer" containerID="5df5183d84ceb81bf0d4376d904f9be9690cc746dce5c397000d67f2d668b404"
Sep 09 20:51:50.514753 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:51:50.514739    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df5183d84ceb81bf0d4376d904f9be9690cc746dce5c397000d67f2d668b404\": container with ID starting with 5df5183d84ceb81bf0d4376d904f9be9690cc746dce5c397000d67f2d668b404 not found: ID does not exist" containerID="5df5183d84ceb81bf0d4376d904f9be9690cc746dce5c397000d67f2d668b404"
Sep 09 20:51:50.514793 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.514758    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df5183d84ceb81bf0d4376d904f9be9690cc746dce5c397000d67f2d668b404"} err="failed to get container status \"5df5183d84ceb81bf0d4376d904f9be9690cc746dce5c397000d67f2d668b404\": rpc error: code = NotFound desc = could not find container \"5df5183d84ceb81bf0d4376d904f9be9690cc746dce5c397000d67f2d668b404\": container with ID starting with 5df5183d84ceb81bf0d4376d904f9be9690cc746dce5c397000d67f2d668b404 not found: ID does not exist"
Sep 09 20:51:50.520497 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.520477    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"]
Sep 09 20:51:50.522557 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.522537    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-999c8b38-openshift-default-64745dd4cb-vqw78"]
Sep 09 20:51:50.578824 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.578802    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-token\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:51:50.578824 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.578822    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istiod-ca-cert\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:51:50.578955 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.578832    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h44p\" (UniqueName: \"kubernetes.io/projected/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-kube-api-access-4h44p\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:51:50.578955 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.578841    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-podinfo\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:51:50.578955 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.578850    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b-istio-envoy\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:51:50.893278 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:50.893250    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" path="/var/lib/kubelet/pods/7c32eaef-e67e-4fef-a1f1-a22abe2dc45b/volumes"
Sep 09 20:51:51.291895 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.291869    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"]
Sep 09 20:51:51.292401 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.292382    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" containerName="istio-proxy"
Sep 09 20:51:51.292401 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.292401    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" containerName="istio-proxy"
Sep 09 20:51:51.292540 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.292414    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" containerName="istio-proxy"
Sep 09 20:51:51.292540 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.292419    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" containerName="istio-proxy"
Sep 09 20:51:51.292540 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.292522    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" containerName="istio-proxy"
Sep 09 20:51:51.295433 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.295417    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:51:51.297912 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.297887    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-933e9d\"/\"distinct-w4-933e9d-kserve-self-signed-certs\""
Sep 09 20:51:51.305720 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.305697    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"]
Sep 09 20:51:51.387398 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.387368    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-dshm\") pod \"distinct-w4-933e9d-kserve-df9fb48cd-p4x77\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") " pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:51:51.387524 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.387407    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd2caf3-add9-49bc-a256-ac962f5b9a69-tls-certs\") pod \"distinct-w4-933e9d-kserve-df9fb48cd-p4x77\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") " pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:51:51.387524 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.387432    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkzs8\" (UniqueName: \"kubernetes.io/projected/ccd2caf3-add9-49bc-a256-ac962f5b9a69-kube-api-access-xkzs8\") pod \"distinct-w4-933e9d-kserve-df9fb48cd-p4x77\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") " pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:51:51.387598 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.387536    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-model-cache\") pod \"distinct-w4-933e9d-kserve-df9fb48cd-p4x77\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") " pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:51:51.387598 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.387584    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-tmp-dir\") pod \"distinct-w4-933e9d-kserve-df9fb48cd-p4x77\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") " pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:51:51.387665 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.387605    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-home\") pod \"distinct-w4-933e9d-kserve-df9fb48cd-p4x77\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") " pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:51:51.488371 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.488338    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-tmp-dir\") pod \"distinct-w4-933e9d-kserve-df9fb48cd-p4x77\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") " pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:51:51.488802 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.488376    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-home\") pod \"distinct-w4-933e9d-kserve-df9fb48cd-p4x77\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") " pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:51:51.488802 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.488461    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-dshm\") pod \"distinct-w4-933e9d-kserve-df9fb48cd-p4x77\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") " pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:51:51.488802 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.488506    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd2caf3-add9-49bc-a256-ac962f5b9a69-tls-certs\") pod \"distinct-w4-933e9d-kserve-df9fb48cd-p4x77\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") " pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:51:51.488802 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.488532    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkzs8\" (UniqueName: \"kubernetes.io/projected/ccd2caf3-add9-49bc-a256-ac962f5b9a69-kube-api-access-xkzs8\") pod \"distinct-w4-933e9d-kserve-df9fb48cd-p4x77\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") " pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:51:51.488802 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.488608    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-model-cache\") pod \"distinct-w4-933e9d-kserve-df9fb48cd-p4x77\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") " pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:51:51.488802 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.488726    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-tmp-dir\") pod \"distinct-w4-933e9d-kserve-df9fb48cd-p4x77\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") " pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:51:51.488802 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.488766    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-home\") pod \"distinct-w4-933e9d-kserve-df9fb48cd-p4x77\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") " pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:51:51.489178 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.488948    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-model-cache\") pod \"distinct-w4-933e9d-kserve-df9fb48cd-p4x77\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") " pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:51:51.490571 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.490551    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-dshm\") pod \"distinct-w4-933e9d-kserve-df9fb48cd-p4x77\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") " pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:51:51.490934 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.490897    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd2caf3-add9-49bc-a256-ac962f5b9a69-tls-certs\") pod \"distinct-w4-933e9d-kserve-df9fb48cd-p4x77\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") " pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:51:51.498448 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.498424    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkzs8\" (UniqueName: \"kubernetes.io/projected/ccd2caf3-add9-49bc-a256-ac962f5b9a69-kube-api-access-xkzs8\") pod \"distinct-w4-933e9d-kserve-df9fb48cd-p4x77\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") " pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:51:51.605712 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.605638    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:51:51.879522 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:51.879496    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"]
Sep 09 20:51:51.881336 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:51:51.881308    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccd2caf3_add9_49bc_a256_ac962f5b9a69.slice/crio-3c8201857136eb9825a1bdbdd764414743b883d8fb0cb2191cfe06f7102f81f1 WatchSource:0}: Error finding container 3c8201857136eb9825a1bdbdd764414743b883d8fb0cb2191cfe06f7102f81f1: Status 404 returned error can't find the container with id 3c8201857136eb9825a1bdbdd764414743b883d8fb0cb2191cfe06f7102f81f1
Sep 09 20:51:52.511022 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:52.510985    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77" event={"ID":"ccd2caf3-add9-49bc-a256-ac962f5b9a69","Type":"ContainerStarted","Data":"7e111343cec27d8917aaf405e28f0e3611107144aa40492fcebc6d8f84bfc8f6"}
Sep 09 20:51:52.511396 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:52.511031    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77" event={"ID":"ccd2caf3-add9-49bc-a256-ac962f5b9a69","Type":"ContainerStarted","Data":"3c8201857136eb9825a1bdbdd764414743b883d8fb0cb2191cfe06f7102f81f1"}
Sep 09 20:51:52.537540 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:51:52.537485    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77" podStartSLOduration=1.537468492 podStartE2EDuration="1.537468492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:51:52.536261916 +0000 UTC m=+1158.216222227" watchObservedRunningTime="2026-09-09 20:51:52.537468492 +0000 UTC m=+1158.217428821"
Sep 09 20:52:01.606466 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:01.606379    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:52:01.606466 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:01.606434    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:52:01.618617 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:01.618593    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:52:02.577594 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:02.577566    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 20:52:03.980537 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:03.980506    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2"]
Sep 09 20:52:03.981223 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:03.981202    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c32eaef-e67e-4fef-a1f1-a22abe2dc45b" containerName="istio-proxy"
Sep 09 20:52:03.983352 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:03.983332    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2"
Sep 09 20:52:03.986403 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:03.986385    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-78a7d5-dockercfg-6x27p\""
Sep 09 20:52:03.986952 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:03.986931    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-78a7d5\""
Sep 09 20:52:03.999043 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:03.995942    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7d8f338-b592-4c38-93d5-d41bbefe9c63-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2\" (UID: \"b7d8f338-b592-4c38-93d5-d41bbefe9c63\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2"
Sep 09 20:52:03.999043 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:03.996184    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndcrg\" (UniqueName: \"kubernetes.io/projected/b7d8f338-b592-4c38-93d5-d41bbefe9c63-kube-api-access-ndcrg\") pod \"payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2\" (UID: \"b7d8f338-b592-4c38-93d5-d41bbefe9c63\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2"
Sep 09 20:52:04.003175 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:04.000444    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2"]
Sep 09 20:52:04.097255 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:04.097217    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7d8f338-b592-4c38-93d5-d41bbefe9c63-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2\" (UID: \"b7d8f338-b592-4c38-93d5-d41bbefe9c63\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2"
Sep 09 20:52:04.097255 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:04.097262    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndcrg\" (UniqueName: \"kubernetes.io/projected/b7d8f338-b592-4c38-93d5-d41bbefe9c63-kube-api-access-ndcrg\") pod \"payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2\" (UID: \"b7d8f338-b592-4c38-93d5-d41bbefe9c63\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2"
Sep 09 20:52:04.097965 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:04.097944    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7d8f338-b592-4c38-93d5-d41bbefe9c63-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2\" (UID: \"b7d8f338-b592-4c38-93d5-d41bbefe9c63\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2"
Sep 09 20:52:04.107943 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:04.107919    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndcrg\" (UniqueName: \"kubernetes.io/projected/b7d8f338-b592-4c38-93d5-d41bbefe9c63-kube-api-access-ndcrg\") pod \"payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2\" (UID: \"b7d8f338-b592-4c38-93d5-d41bbefe9c63\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2"
Sep 09 20:52:04.294761 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:04.294733    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2"
Sep 09 20:52:04.567205 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:04.567180    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2"]
Sep 09 20:52:04.569637 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:52:04.569605    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d8f338_b592_4c38_93d5_d41bbefe9c63.slice/crio-1135ba3880314decf932f2c44948df4883b6ffcadc62ed9bc19e600993694aba WatchSource:0}: Error finding container 1135ba3880314decf932f2c44948df4883b6ffcadc62ed9bc19e600993694aba: Status 404 returned error can't find the container with id 1135ba3880314decf932f2c44948df4883b6ffcadc62ed9bc19e600993694aba
Sep 09 20:52:04.579156 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:04.579132    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2" event={"ID":"b7d8f338-b592-4c38-93d5-d41bbefe9c63","Type":"ContainerStarted","Data":"1135ba3880314decf932f2c44948df4883b6ffcadc62ed9bc19e600993694aba"}
Sep 09 20:52:05.586179 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:05.586141    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2" event={"ID":"b7d8f338-b592-4c38-93d5-d41bbefe9c63","Type":"ContainerStarted","Data":"8a9504e02d239c3e7af826456f095c236bb38b52fa43d79f794d30038113617f"}
Sep 09 20:52:05.586537 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:05.586220    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2"
Sep 09 20:52:05.604727 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:05.604682    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2" podStartSLOduration=2.604668332 podStartE2EDuration="2.604668332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:05.602610448 +0000 UTC m=+1171.282570757" watchObservedRunningTime="2026-09-09 20:52:05.604668332 +0000 UTC m=+1171.284628680"
Sep 09 20:52:16.594069 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:16.594040    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2"
Sep 09 20:52:24.397526 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:24.397488    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:52:25.156251 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:25.156218    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg"]
Sep 09 20:52:25.156488 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:25.156451    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg" podUID="92075b8b-5763-446a-ae3c-7f2ddc4124b7" containerName="payload-processing" containerID="cri-o://7b7b8b90a97075bc1bb972716987bd41cb56c7e22a5815858639e90234d4e47d" gracePeriod=30
Sep 09 20:52:25.413650 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:25.413598    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg"
Sep 09 20:52:25.486674 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:25.486646    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jkjh\" (UniqueName: \"kubernetes.io/projected/92075b8b-5763-446a-ae3c-7f2ddc4124b7-kube-api-access-9jkjh\") pod \"92075b8b-5763-446a-ae3c-7f2ddc4124b7\" (UID: \"92075b8b-5763-446a-ae3c-7f2ddc4124b7\") "
Sep 09 20:52:25.486821 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:25.486771    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92075b8b-5763-446a-ae3c-7f2ddc4124b7-plugins-config-volume\") pod \"92075b8b-5763-446a-ae3c-7f2ddc4124b7\" (UID: \"92075b8b-5763-446a-ae3c-7f2ddc4124b7\") "
Sep 09 20:52:25.487148 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:25.487125    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92075b8b-5763-446a-ae3c-7f2ddc4124b7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "92075b8b-5763-446a-ae3c-7f2ddc4124b7" (UID: "92075b8b-5763-446a-ae3c-7f2ddc4124b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:52:25.488690 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:25.488669    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92075b8b-5763-446a-ae3c-7f2ddc4124b7-kube-api-access-9jkjh" (OuterVolumeSpecName: "kube-api-access-9jkjh") pod "92075b8b-5763-446a-ae3c-7f2ddc4124b7" (UID: "92075b8b-5763-446a-ae3c-7f2ddc4124b7"). InnerVolumeSpecName "kube-api-access-9jkjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:52:25.587777 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:25.587742    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92075b8b-5763-446a-ae3c-7f2ddc4124b7-plugins-config-volume\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:52:25.587777 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:25.587767    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jkjh\" (UniqueName: \"kubernetes.io/projected/92075b8b-5763-446a-ae3c-7f2ddc4124b7-kube-api-access-9jkjh\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:52:25.696850 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:25.696776    2556 generic.go:356] "Generic (PLEG): container finished" podID="92075b8b-5763-446a-ae3c-7f2ddc4124b7" containerID="7b7b8b90a97075bc1bb972716987bd41cb56c7e22a5815858639e90234d4e47d" exitCode=0
Sep 09 20:52:25.696850 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:25.696820    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg" event={"ID":"92075b8b-5763-446a-ae3c-7f2ddc4124b7","Type":"ContainerDied","Data":"7b7b8b90a97075bc1bb972716987bd41cb56c7e22a5815858639e90234d4e47d"}
Sep 09 20:52:25.697016 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:25.696856    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg" event={"ID":"92075b8b-5763-446a-ae3c-7f2ddc4124b7","Type":"ContainerDied","Data":"d217aa6e96101da95d01417de40991ef1cc6df8725c341dac4a1850fb8cc4388"}
Sep 09 20:52:25.697016 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:25.696879    2556 scope.go:117] "RemoveContainer" containerID="7b7b8b90a97075bc1bb972716987bd41cb56c7e22a5815858639e90234d4e47d"
Sep 09 20:52:25.697016 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:25.696821    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg"
Sep 09 20:52:25.706894 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:25.706876    2556 scope.go:117] "RemoveContainer" containerID="7b7b8b90a97075bc1bb972716987bd41cb56c7e22a5815858639e90234d4e47d"
Sep 09 20:52:25.707187 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:52:25.707168    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b7b8b90a97075bc1bb972716987bd41cb56c7e22a5815858639e90234d4e47d\": container with ID starting with 7b7b8b90a97075bc1bb972716987bd41cb56c7e22a5815858639e90234d4e47d not found: ID does not exist" containerID="7b7b8b90a97075bc1bb972716987bd41cb56c7e22a5815858639e90234d4e47d"
Sep 09 20:52:25.707237 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:25.707195    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b7b8b90a97075bc1bb972716987bd41cb56c7e22a5815858639e90234d4e47d"} err="failed to get container status \"7b7b8b90a97075bc1bb972716987bd41cb56c7e22a5815858639e90234d4e47d\": rpc error: code = NotFound desc = could not find container \"7b7b8b90a97075bc1bb972716987bd41cb56c7e22a5815858639e90234d4e47d\": container with ID starting with 7b7b8b90a97075bc1bb972716987bd41cb56c7e22a5815858639e90234d4e47d not found: ID does not exist"
Sep 09 20:52:25.720467 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:25.720438    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg"]
Sep 09 20:52:25.724261 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:25.724232    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2d6f6c-6b66d99bf7-n94lg"]
Sep 09 20:52:26.892655 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:26.892616    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92075b8b-5763-446a-ae3c-7f2ddc4124b7" path="/var/lib/kubelet/pods/92075b8b-5763-446a-ae3c-7f2ddc4124b7/volumes"
Sep 09 20:52:27.524581 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.524547    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"]
Sep 09 20:52:27.525228 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.525205    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92075b8b-5763-446a-ae3c-7f2ddc4124b7" containerName="payload-processing"
Sep 09 20:52:27.525368 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.525287    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="92075b8b-5763-446a-ae3c-7f2ddc4124b7" containerName="payload-processing"
Sep 09 20:52:27.525452 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.525437    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="92075b8b-5763-446a-ae3c-7f2ddc4124b7" containerName="payload-processing"
Sep 09 20:52:27.529852 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.529825    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.532751 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.532724    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-3fa05f-openshift-default-dockercfg-fbd9n\""
Sep 09 20:52:27.542973 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.542949    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"]
Sep 09 20:52:27.606987 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.606953    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-workload-certs\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.607112 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.607063    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls4n8\" (UniqueName: \"kubernetes.io/projected/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-kube-api-access-ls4n8\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.607112 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.607102    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-data\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.607209 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.607159    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-credential-socket\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.607209 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.607184    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-podinfo\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.607320 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.607287    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-envoy\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.607376 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.607332    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-token\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.607491 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.607473    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-workload-socket\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.607536 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.607504    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istiod-ca-cert\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.708868 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.708829    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-workload-socket\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.708868 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.708869    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istiod-ca-cert\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.709129 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.708934    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-workload-certs\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.709129 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.708972    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls4n8\" (UniqueName: \"kubernetes.io/projected/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-kube-api-access-ls4n8\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.709129 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.708991    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-data\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.709129 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.709017    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-credential-socket\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.709129 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.709032    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-podinfo\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.709129 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.709089    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-envoy\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.709129 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.709127    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-token\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.709475 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.709403    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-workload-socket\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.709475 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.709429    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-credential-socket\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.709559 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.709531    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-workload-certs\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.709699 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.709671    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-data\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.710114 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.710092    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istiod-ca-cert\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.711812 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.711786    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-envoy\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.712190 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.712171    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-podinfo\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.719554 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.719529    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls4n8\" (UniqueName: \"kubernetes.io/projected/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-kube-api-access-ls4n8\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.719661 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.719644    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-token\") pod \"e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") " pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:27.843317 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:27.843231    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:28.155324 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:28.155290    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"]
Sep 09 20:52:28.157807 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:52:28.157782    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc928d9_3ea3_46bd_a1d1_def8adefab3d.slice/crio-ae672042a4874c4487f81121b5dd784ed710fbb5cae2b849cce03d173b205176 WatchSource:0}: Error finding container ae672042a4874c4487f81121b5dd784ed710fbb5cae2b849cce03d173b205176: Status 404 returned error can't find the container with id ae672042a4874c4487f81121b5dd784ed710fbb5cae2b849cce03d173b205176
Sep 09 20:52:28.160173 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:28.160142    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:52:28.160248 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:28.160204    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:52:28.160248 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:28.160231    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:52:28.724119 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:28.724079    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn" event={"ID":"5fc928d9-3ea3-46bd-a1d1-def8adefab3d","Type":"ContainerStarted","Data":"3b56a080d72573dacf3980751b790a5dde8a78addc97baace6966c386023e576"}
Sep 09 20:52:28.724493 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:28.724134    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn" event={"ID":"5fc928d9-3ea3-46bd-a1d1-def8adefab3d","Type":"ContainerStarted","Data":"ae672042a4874c4487f81121b5dd784ed710fbb5cae2b849cce03d173b205176"}
Sep 09 20:52:28.747402 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:28.747351    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn" podStartSLOduration=1.747337503 podStartE2EDuration="1.747337503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:28.745599905 +0000 UTC m=+1194.425560227" watchObservedRunningTime="2026-09-09 20:52:28.747337503 +0000 UTC m=+1194.427297813"
Sep 09 20:52:28.843549 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:28.843521    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:29.849040 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:29.849012    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:30.735775 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:30.735742    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:30.736843 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:30.736817    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:44.781416 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.781382    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"]
Sep 09 20:52:44.786652 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.786631    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:44.790961 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.790936    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2b2343\"/\"kube-root-ca.crt\""
Sep 09 20:52:44.791114 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.791016    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2b2343\"/\"openshift-service-ca.crt\""
Sep 09 20:52:44.791235 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.791037    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2b2343\"/\"distinct-w0-2b2343-kserve-self-signed-certs\""
Sep 09 20:52:44.791444 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.791424    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2b2343\"/\"default-dockercfg-v4nrn\""
Sep 09 20:52:44.794347 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.794330    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"]
Sep 09 20:52:44.867543 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.867520    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-dshm\") pod \"distinct-w0-2b2343-kserve-f7f7f778f-s72g2\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") " pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:44.867655 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.867555    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-home\") pod \"distinct-w0-2b2343-kserve-f7f7f778f-s72g2\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") " pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:44.867655 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.867577    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-model-cache\") pod \"distinct-w0-2b2343-kserve-f7f7f778f-s72g2\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") " pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:44.867734 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.867679    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55619605-52ef-4109-8856-d64d81a6d8d4-tls-certs\") pod \"distinct-w0-2b2343-kserve-f7f7f778f-s72g2\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") " pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:44.867771 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.867759    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-tmp-dir\") pod \"distinct-w0-2b2343-kserve-f7f7f778f-s72g2\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") " pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:44.867815 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.867803    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99cdb\" (UniqueName: \"kubernetes.io/projected/55619605-52ef-4109-8856-d64d81a6d8d4-kube-api-access-99cdb\") pod \"distinct-w0-2b2343-kserve-f7f7f778f-s72g2\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") " pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:44.968641 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.968610    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-tmp-dir\") pod \"distinct-w0-2b2343-kserve-f7f7f778f-s72g2\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") " pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:44.968785 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.968668    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99cdb\" (UniqueName: \"kubernetes.io/projected/55619605-52ef-4109-8856-d64d81a6d8d4-kube-api-access-99cdb\") pod \"distinct-w0-2b2343-kserve-f7f7f778f-s72g2\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") " pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:44.968785 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.968690    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-dshm\") pod \"distinct-w0-2b2343-kserve-f7f7f778f-s72g2\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") " pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:44.968785 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.968721    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-home\") pod \"distinct-w0-2b2343-kserve-f7f7f778f-s72g2\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") " pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:44.968785 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.968742    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-model-cache\") pod \"distinct-w0-2b2343-kserve-f7f7f778f-s72g2\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") " pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:44.969012 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.968810    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55619605-52ef-4109-8856-d64d81a6d8d4-tls-certs\") pod \"distinct-w0-2b2343-kserve-f7f7f778f-s72g2\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") " pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:44.969139 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.969116    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-tmp-dir\") pod \"distinct-w0-2b2343-kserve-f7f7f778f-s72g2\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") " pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:44.969206 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.969181    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-model-cache\") pod \"distinct-w0-2b2343-kserve-f7f7f778f-s72g2\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") " pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:44.969277 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.969252    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-home\") pod \"distinct-w0-2b2343-kserve-f7f7f778f-s72g2\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") " pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:44.971142 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.971117    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-dshm\") pod \"distinct-w0-2b2343-kserve-f7f7f778f-s72g2\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") " pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:44.971418 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.971396    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55619605-52ef-4109-8856-d64d81a6d8d4-tls-certs\") pod \"distinct-w0-2b2343-kserve-f7f7f778f-s72g2\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") " pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:44.978150 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:44.978126    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99cdb\" (UniqueName: \"kubernetes.io/projected/55619605-52ef-4109-8856-d64d81a6d8d4-kube-api-access-99cdb\") pod \"distinct-w0-2b2343-kserve-f7f7f778f-s72g2\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") " pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:45.098185 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:45.098108    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:45.375705 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:45.375674    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"]
Sep 09 20:52:45.377618 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:52:45.377584    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55619605_52ef_4109_8856_d64d81a6d8d4.slice/crio-cd34aede7d16efc79114a99aba69d69569a624437e05348b545979ce1689c93e WatchSource:0}: Error finding container cd34aede7d16efc79114a99aba69d69569a624437e05348b545979ce1689c93e: Status 404 returned error can't find the container with id cd34aede7d16efc79114a99aba69d69569a624437e05348b545979ce1689c93e
Sep 09 20:52:45.821141 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:45.821109    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2" event={"ID":"55619605-52ef-4109-8856-d64d81a6d8d4","Type":"ContainerStarted","Data":"c95406e4e5276c8722c61def0c37c5c336493e70308e929d2738d551368b951d"}
Sep 09 20:52:45.821579 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:45.821157    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2" event={"ID":"55619605-52ef-4109-8856-d64d81a6d8d4","Type":"ContainerStarted","Data":"cd34aede7d16efc79114a99aba69d69569a624437e05348b545979ce1689c93e"}
Sep 09 20:52:45.841269 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:45.841218    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2" podStartSLOduration=1.841202282 podStartE2EDuration="1.841202282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:45.83877571 +0000 UTC m=+1211.518736019" watchObservedRunningTime="2026-09-09 20:52:45.841202282 +0000 UTC m=+1211.521162585"
Sep 09 20:52:54.878590 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:54.878553    2556 generic.go:356] "Generic (PLEG): container finished" podID="5fc928d9-3ea3-46bd-a1d1-def8adefab3d" containerID="3b56a080d72573dacf3980751b790a5dde8a78addc97baace6966c386023e576" exitCode=0
Sep 09 20:52:54.878590 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:54.878578    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn" event={"ID":"5fc928d9-3ea3-46bd-a1d1-def8adefab3d","Type":"ContainerDied","Data":"3b56a080d72573dacf3980751b790a5dde8a78addc97baace6966c386023e576"}
Sep 09 20:52:54.879095 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:54.879039    2556 scope.go:117] "RemoveContainer" containerID="3b56a080d72573dacf3980751b790a5dde8a78addc97baace6966c386023e576"
Sep 09 20:52:54.880204 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:54.880179    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:52:54.880286 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:54.880243    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:52:54.880286 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:54.880273    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:52:55.099255 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:55.099221    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:55.099423 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:55.099280    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:55.111886 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:55.111860    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:55.893668 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:55.893624    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn" event={"ID":"5fc928d9-3ea3-46bd-a1d1-def8adefab3d","Type":"ContainerStarted","Data":"4711f7e472deca428d5c30c98d91604af908c2a2b76286c7f85946ebcc648be6"}
Sep 09 20:52:55.905169 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:55.905136    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:52:56.843793 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:56.843759    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:52:56.844872 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:56.844839    2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 09 20:52:56.845048 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:56.844897    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn" podUID="5fc928d9-3ea3-46bd-a1d1-def8adefab3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 09 20:52:57.844365 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:57.844328    2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 09 20:52:57.844749 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:57.844384    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn" podUID="5fc928d9-3ea3-46bd-a1d1-def8adefab3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 09 20:52:58.844399 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:58.844363    2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 09 20:52:58.844800 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:58.844421    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn" podUID="5fc928d9-3ea3-46bd-a1d1-def8adefab3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 09 20:52:59.844422 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:59.844388    2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 09 20:52:59.844816 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:52:59.844453    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn" podUID="5fc928d9-3ea3-46bd-a1d1-def8adefab3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 09 20:53:00.606437 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:00.606401    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:53:00.736107 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:00.736080    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:53:00.843828 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:00.843789    2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 09 20:53:00.844015 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:00.843846    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn" podUID="5fc928d9-3ea3-46bd-a1d1-def8adefab3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 09 20:53:01.847836 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:01.847805    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:53:01.848951 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:01.848930    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 20:53:07.860040 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:07.860008    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"]
Sep 09 20:53:07.860534 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:07.860215    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8" podUID="fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" containerName="istio-proxy" containerID="cri-o://637ef1e4644cfefb3dc6b6c5400d66dff8367d8a3275a126898148cbc68944a4" gracePeriod=30
Sep 09 20:53:08.752038 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:08.752002    2556 patch_prober.go:28] interesting pod/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 20:53:08.752210 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:08.752054    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8" podUID="fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:53:12.992757 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:12.992724    2556 generic.go:356] "Generic (PLEG): container finished" podID="fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" containerID="637ef1e4644cfefb3dc6b6c5400d66dff8367d8a3275a126898148cbc68944a4" exitCode=0
Sep 09 20:53:12.993129 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:12.992800    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8" event={"ID":"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1","Type":"ContainerDied","Data":"637ef1e4644cfefb3dc6b6c5400d66dff8367d8a3275a126898148cbc68944a4"}
Sep 09 20:53:12.993129 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:12.992859    2556 scope.go:117] "RemoveContainer" containerID="362909a3f9a539c039ae11c66046f8fba735bd63f1e9494ff7852fb8740caecd"
Sep 09 20:53:13.118622 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.118599    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:53:13.263597 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.263576    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-token\") pod \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") "
Sep 09 20:53:13.263747 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.263609    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-data\") pod \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") "
Sep 09 20:53:13.263747 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.263648    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-podinfo\") pod \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") "
Sep 09 20:53:13.263747 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.263666    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-credential-socket\") pod \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") "
Sep 09 20:53:13.263747 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.263710    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-envoy\") pod \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") "
Sep 09 20:53:13.263747 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.263731    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz2bx\" (UniqueName: \"kubernetes.io/projected/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-kube-api-access-pz2bx\") pod \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") "
Sep 09 20:53:13.264042 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.263780    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-workload-certs\") pod \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") "
Sep 09 20:53:13.264042 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.263858    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istiod-ca-cert\") pod \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") "
Sep 09 20:53:13.264042 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.263892    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-workload-socket\") pod \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\" (UID: \"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1\") "
Sep 09 20:53:13.264042 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.263958    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-data" (OuterVolumeSpecName: "istio-data") pod "fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" (UID: "fe0a3043-56fc-4f66-bd1b-fca0fb23eef1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:53:13.264261 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.264065    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" (UID: "fe0a3043-56fc-4f66-bd1b-fca0fb23eef1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:53:13.264261 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.264116    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" (UID: "fe0a3043-56fc-4f66-bd1b-fca0fb23eef1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:53:13.264367 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.264280    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" (UID: "fe0a3043-56fc-4f66-bd1b-fca0fb23eef1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:53:13.264367 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.264304    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" (UID: "fe0a3043-56fc-4f66-bd1b-fca0fb23eef1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:53:13.264443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.264387    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-workload-socket\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:53:13.264443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.264408    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-data\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:53:13.264443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.264424    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-credential-socket\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:53:13.264443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.264440    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-workload-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:53:13.265842 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.265817    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" (UID: "fe0a3043-56fc-4f66-bd1b-fca0fb23eef1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 20:53:13.265951 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.265928    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-kube-api-access-pz2bx" (OuterVolumeSpecName: "kube-api-access-pz2bx") pod "fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" (UID: "fe0a3043-56fc-4f66-bd1b-fca0fb23eef1"). InnerVolumeSpecName "kube-api-access-pz2bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:53:13.266263 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.266243    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" (UID: "fe0a3043-56fc-4f66-bd1b-fca0fb23eef1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:53:13.266321 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.266257    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-token" (OuterVolumeSpecName: "istio-token") pod "fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" (UID: "fe0a3043-56fc-4f66-bd1b-fca0fb23eef1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:53:13.365148 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.365125    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istiod-ca-cert\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:53:13.365148 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.365147    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-token\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:53:13.365287 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.365156    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-podinfo\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:53:13.365287 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.365164    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-istio-envoy\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:53:13.365287 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.365173    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz2bx\" (UniqueName: \"kubernetes.io/projected/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1-kube-api-access-pz2bx\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:53:13.999480 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.999448    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"
Sep 09 20:53:13.999922 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.999483    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8" event={"ID":"fe0a3043-56fc-4f66-bd1b-fca0fb23eef1","Type":"ContainerDied","Data":"f29306e5e1cbece4637bfb679fad32322f0334ebc8d8d083f62f17353535a6f0"}
Sep 09 20:53:13.999922 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:13.999534    2556 scope.go:117] "RemoveContainer" containerID="637ef1e4644cfefb3dc6b6c5400d66dff8367d8a3275a126898148cbc68944a4"
Sep 09 20:53:14.035184 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:14.035161    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"]
Sep 09 20:53:14.044597 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:14.044573    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-2d6f6c-openshift-default-56f496fc48-gxvl8"]
Sep 09 20:53:14.894458 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:14.894426    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" path="/var/lib/kubelet/pods/fe0a3043-56fc-4f66-bd1b-fca0fb23eef1/volumes"
Sep 09 20:53:15.595380 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.595344    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l"]
Sep 09 20:53:15.595803 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.595783    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" containerName="istio-proxy"
Sep 09 20:53:15.595803 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.595794    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" containerName="istio-proxy"
Sep 09 20:53:15.595882 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.595876    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" containerName="istio-proxy"
Sep 09 20:53:15.595948 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.595886    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" containerName="istio-proxy"
Sep 09 20:53:15.600381 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.600360    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l"
Sep 09 20:53:15.602816 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.602790    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fbe7be47\""
Sep 09 20:53:15.609630 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.609606    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l"]
Sep 09 20:53:15.623219 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.623186    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf"]
Sep 09 20:53:15.623598 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.623580    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" containerName="istio-proxy"
Sep 09 20:53:15.623598 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.623596    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe0a3043-56fc-4f66-bd1b-fca0fb23eef1" containerName="istio-proxy"
Sep 09 20:53:15.626755 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.626737    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf"
Sep 09 20:53:15.629484 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.629467    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fbe7be47-dockercfg-flbqp\""
Sep 09 20:53:15.637593 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.637574    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf"]
Sep 09 20:53:15.789698 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.789667    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l\" (UID: \"ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l"
Sep 09 20:53:15.789881 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.789713    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pg2k\" (UniqueName: \"kubernetes.io/projected/01483de9-50af-4cbc-8d3e-373e8a6fc1f8-kube-api-access-7pg2k\") pod \"payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf\" (UID: \"01483de9-50af-4cbc-8d3e-373e8a6fc1f8\") " pod="openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf"
Sep 09 20:53:15.789881 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.789853    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg7cz\" (UniqueName: \"kubernetes.io/projected/ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9-kube-api-access-cg7cz\") pod \"payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l\" (UID: \"ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l"
Sep 09 20:53:15.790032 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.789886    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01483de9-50af-4cbc-8d3e-373e8a6fc1f8-plugins-config-volume\") pod \"payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf\" (UID: \"01483de9-50af-4cbc-8d3e-373e8a6fc1f8\") " pod="openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf"
Sep 09 20:53:15.890797 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.890712    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l\" (UID: \"ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l"
Sep 09 20:53:15.890797 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.890762    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pg2k\" (UniqueName: \"kubernetes.io/projected/01483de9-50af-4cbc-8d3e-373e8a6fc1f8-kube-api-access-7pg2k\") pod \"payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf\" (UID: \"01483de9-50af-4cbc-8d3e-373e8a6fc1f8\") " pod="openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf"
Sep 09 20:53:15.891047 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.890864    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7cz\" (UniqueName: \"kubernetes.io/projected/ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9-kube-api-access-cg7cz\") pod \"payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l\" (UID: \"ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l"
Sep 09 20:53:15.891047 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.890893    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01483de9-50af-4cbc-8d3e-373e8a6fc1f8-plugins-config-volume\") pod \"payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf\" (UID: \"01483de9-50af-4cbc-8d3e-373e8a6fc1f8\") " pod="openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf"
Sep 09 20:53:15.891474 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.891450    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l\" (UID: \"ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l"
Sep 09 20:53:15.891630 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.891609    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01483de9-50af-4cbc-8d3e-373e8a6fc1f8-plugins-config-volume\") pod \"payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf\" (UID: \"01483de9-50af-4cbc-8d3e-373e8a6fc1f8\") " pod="openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf"
Sep 09 20:53:15.899380 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.899353    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg7cz\" (UniqueName: \"kubernetes.io/projected/ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9-kube-api-access-cg7cz\") pod \"payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l\" (UID: \"ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l"
Sep 09 20:53:15.899494 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.899406    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pg2k\" (UniqueName: \"kubernetes.io/projected/01483de9-50af-4cbc-8d3e-373e8a6fc1f8-kube-api-access-7pg2k\") pod \"payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf\" (UID: \"01483de9-50af-4cbc-8d3e-373e8a6fc1f8\") " pod="openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf"
Sep 09 20:53:15.912096 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.912078    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l"
Sep 09 20:53:15.936875 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:15.936832    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf"
Sep 09 20:53:16.187142 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:16.187116    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l"]
Sep 09 20:53:16.188715 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:53:16.188691    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9a7fc7_8c1e_4f75_b927_9b685f81a4a9.slice/crio-4d609a2212380eb63d308b56d42f467ddcf0dbf8d10dee1407d48b0fcdcd2862 WatchSource:0}: Error finding container 4d609a2212380eb63d308b56d42f467ddcf0dbf8d10dee1407d48b0fcdcd2862: Status 404 returned error can't find the container with id 4d609a2212380eb63d308b56d42f467ddcf0dbf8d10dee1407d48b0fcdcd2862
Sep 09 20:53:16.221502 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:16.221480    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf"]
Sep 09 20:53:16.222958 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:53:16.222932    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01483de9_50af_4cbc_8d3e_373e8a6fc1f8.slice/crio-c0fa7413c1e19411b73096c860803d9aead47fba6f7e76658b6aa2a09816f27d WatchSource:0}: Error finding container c0fa7413c1e19411b73096c860803d9aead47fba6f7e76658b6aa2a09816f27d: Status 404 returned error can't find the container with id c0fa7413c1e19411b73096c860803d9aead47fba6f7e76658b6aa2a09816f27d
Sep 09 20:53:17.031047 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:17.031009    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l" event={"ID":"ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9","Type":"ContainerStarted","Data":"4fdf591f4c29c0a153f0fe318838d16a00afba7f8c39a4e646175d94109ef162"}
Sep 09 20:53:17.031477 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:17.031103    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l"
Sep 09 20:53:17.031477 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:17.031132    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l" event={"ID":"ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9","Type":"ContainerStarted","Data":"4d609a2212380eb63d308b56d42f467ddcf0dbf8d10dee1407d48b0fcdcd2862"}
Sep 09 20:53:17.032529 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:17.032505    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf" event={"ID":"01483de9-50af-4cbc-8d3e-373e8a6fc1f8","Type":"ContainerStarted","Data":"a34733430582e04c45975c5d5ad4912c8838a3f12ecdb4cd9289b6aa43ec2d75"}
Sep 09 20:53:17.032649 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:17.032543    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf" event={"ID":"01483de9-50af-4cbc-8d3e-373e8a6fc1f8","Type":"ContainerStarted","Data":"c0fa7413c1e19411b73096c860803d9aead47fba6f7e76658b6aa2a09816f27d"}
Sep 09 20:53:17.032649 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:17.032616    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf"
Sep 09 20:53:17.058529 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:17.058478    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l" podStartSLOduration=2.058460305 podStartE2EDuration="2.058460305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:17.05598422 +0000 UTC m=+1242.735944539" watchObservedRunningTime="2026-09-09 20:53:17.058460305 +0000 UTC m=+1242.738420616"
Sep 09 20:53:17.075314 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:17.075271    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf" podStartSLOduration=2.075257812 podStartE2EDuration="2.075257812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:17.074372634 +0000 UTC m=+1242.754332946" watchObservedRunningTime="2026-09-09 20:53:17.075257812 +0000 UTC m=+1242.755218121"
Sep 09 20:53:28.040738 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:28.040662    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l"
Sep 09 20:53:28.041213 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:28.040845    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf"
Sep 09 20:53:32.807593 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:32.807547    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:53:34.818924 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:34.818876    2556 scope.go:117] "RemoveContainer" containerID="63b522ba45ceff81381fbe72604b5216db5268366bf36bae5113cf817e3a534a"
Sep 09 20:53:34.827958 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:34.827933    2556 scope.go:117] "RemoveContainer" containerID="103bb1fa63a015abf792c9f26a03b8ea05d1696b8deb72921850e9a9bf5a96c8"
Sep 09 20:53:34.835950 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:34.835932    2556 scope.go:117] "RemoveContainer" containerID="557dfcf2cc32bb2d5fd3199ecf969f0692bee36e0f84e7aa99003771c2be1338"
Sep 09 20:53:44.578299 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.578263    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"]
Sep 09 20:53:44.583547 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.583518    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:53:44.587669 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.587644    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-a214cb\"/\"openshift-service-ca.crt\""
Sep 09 20:53:44.587824 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.587646    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-a214cb\"/\"noten-model-a214cb-kserve-self-signed-certs\""
Sep 09 20:53:44.587960 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.587652    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-a214cb\"/\"kube-root-ca.crt\""
Sep 09 20:53:44.588074 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.587711    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-a214cb\"/\"default-dockercfg-djxwv\""
Sep 09 20:53:44.591472 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.591451    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"]
Sep 09 20:53:44.644422 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.644395    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dcf6537-1246-4938-80c1-77cdbb93a66d-tls-certs\") pod \"noten-model-a214cb-kserve-76dcbd6c5-7tmrm\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") " pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:53:44.644554 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.644446    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-home\") pod \"noten-model-a214cb-kserve-76dcbd6c5-7tmrm\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") " pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:53:44.644554 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.644536    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-tmp-dir\") pod \"noten-model-a214cb-kserve-76dcbd6c5-7tmrm\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") " pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:53:44.644646 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.644570    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-dshm\") pod \"noten-model-a214cb-kserve-76dcbd6c5-7tmrm\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") " pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:53:44.644646 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.644606    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95blv\" (UniqueName: \"kubernetes.io/projected/5dcf6537-1246-4938-80c1-77cdbb93a66d-kube-api-access-95blv\") pod \"noten-model-a214cb-kserve-76dcbd6c5-7tmrm\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") " pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:53:44.644753 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.644697    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-model-cache\") pod \"noten-model-a214cb-kserve-76dcbd6c5-7tmrm\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") " pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:53:44.745895 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.745869    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dcf6537-1246-4938-80c1-77cdbb93a66d-tls-certs\") pod \"noten-model-a214cb-kserve-76dcbd6c5-7tmrm\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") " pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:53:44.746035 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.745957    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-home\") pod \"noten-model-a214cb-kserve-76dcbd6c5-7tmrm\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") " pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:53:44.746035 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.746012    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-tmp-dir\") pod \"noten-model-a214cb-kserve-76dcbd6c5-7tmrm\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") " pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:53:44.746116 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.746045    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-dshm\") pod \"noten-model-a214cb-kserve-76dcbd6c5-7tmrm\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") " pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:53:44.746116 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.746093    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95blv\" (UniqueName: \"kubernetes.io/projected/5dcf6537-1246-4938-80c1-77cdbb93a66d-kube-api-access-95blv\") pod \"noten-model-a214cb-kserve-76dcbd6c5-7tmrm\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") " pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:53:44.746224 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.746154    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-model-cache\") pod \"noten-model-a214cb-kserve-76dcbd6c5-7tmrm\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") " pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:53:44.746394 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.746370    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-home\") pod \"noten-model-a214cb-kserve-76dcbd6c5-7tmrm\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") " pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:53:44.746507 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.746423    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-tmp-dir\") pod \"noten-model-a214cb-kserve-76dcbd6c5-7tmrm\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") " pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:53:44.746553 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.746502    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-model-cache\") pod \"noten-model-a214cb-kserve-76dcbd6c5-7tmrm\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") " pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:53:44.748151 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.748124    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-dshm\") pod \"noten-model-a214cb-kserve-76dcbd6c5-7tmrm\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") " pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:53:44.748312 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.748295    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dcf6537-1246-4938-80c1-77cdbb93a66d-tls-certs\") pod \"noten-model-a214cb-kserve-76dcbd6c5-7tmrm\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") " pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:53:44.754837 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.754818    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95blv\" (UniqueName: \"kubernetes.io/projected/5dcf6537-1246-4938-80c1-77cdbb93a66d-kube-api-access-95blv\") pod \"noten-model-a214cb-kserve-76dcbd6c5-7tmrm\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") " pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:53:44.894984 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:44.894901    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:53:45.180549 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:45.180522    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"]
Sep 09 20:53:45.181663 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:53:45.181627    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5dcf6537_1246_4938_80c1_77cdbb93a66d.slice/crio-ccdbb2f47b6c787046ebb34c4367d5d9a333a877961e8b00c53f7dbaf42e6b60 WatchSource:0}: Error finding container ccdbb2f47b6c787046ebb34c4367d5d9a333a877961e8b00c53f7dbaf42e6b60: Status 404 returned error can't find the container with id ccdbb2f47b6c787046ebb34c4367d5d9a333a877961e8b00c53f7dbaf42e6b60
Sep 09 20:53:45.198441 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:45.198411    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm" event={"ID":"5dcf6537-1246-4938-80c1-77cdbb93a66d","Type":"ContainerStarted","Data":"ccdbb2f47b6c787046ebb34c4367d5d9a333a877961e8b00c53f7dbaf42e6b60"}
Sep 09 20:53:46.206142 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:46.206108    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm" event={"ID":"5dcf6537-1246-4938-80c1-77cdbb93a66d","Type":"ContainerStarted","Data":"dd68182313155d20d96f244a729d1c1e181e16580d7c0efe5f2562e7f5676e1a"}
Sep 09 20:53:46.227433 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:46.227385    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm" podStartSLOduration=2.227370013 podStartE2EDuration="2.227370013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:46.224590782 +0000 UTC m=+1271.904551096" watchObservedRunningTime="2026-09-09 20:53:46.227370013 +0000 UTC m=+1271.907330386"
Sep 09 20:53:54.895500 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:54.895474    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:53:54.895876 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:54.895516    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:53:54.907762 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:54.907734    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:53:55.266328 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:53:55.266303    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:54:00.203684 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:00.203651    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:54:02.864207 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:02.864179    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7643f-w28nz"]
Sep 09 20:54:02.869196 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:02.869173    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7643f-w28nz"
Sep 09 20:54:02.871923 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:02.871884    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsks9\""
Sep 09 20:54:02.876628 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:02.876606    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7643f-w28nz"]
Sep 09 20:54:03.018278 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:03.018250    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dca9bd6-70e8-443a-91fb-b07c84d3e9f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c7643f-w28nz\" (UID: \"0dca9bd6-70e8-443a-91fb-b07c84d3e9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7643f-w28nz"
Sep 09 20:54:03.119549 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:03.119474    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dca9bd6-70e8-443a-91fb-b07c84d3e9f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c7643f-w28nz\" (UID: \"0dca9bd6-70e8-443a-91fb-b07c84d3e9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7643f-w28nz"
Sep 09 20:54:03.120098 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:03.120078    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dca9bd6-70e8-443a-91fb-b07c84d3e9f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c7643f-w28nz\" (UID: \"0dca9bd6-70e8-443a-91fb-b07c84d3e9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7643f-w28nz"
Sep 09 20:54:03.182411 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:03.182388    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7643f-w28nz"
Sep 09 20:54:03.455171 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:03.455147    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7643f-w28nz"]
Sep 09 20:54:03.457070 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:54:03.457043    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dca9bd6_70e8_443a_91fb_b07c84d3e9f9.slice/crio-5622894d4d8dee80bac9492e8ba8577ec438144df50b0d8d10494c2f6d6f5435 WatchSource:0}: Error finding container 5622894d4d8dee80bac9492e8ba8577ec438144df50b0d8d10494c2f6d6f5435: Status 404 returned error can't find the container with id 5622894d4d8dee80bac9492e8ba8577ec438144df50b0d8d10494c2f6d6f5435
Sep 09 20:54:04.305120 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:04.305089    2556 generic.go:356] "Generic (PLEG): container finished" podID="0dca9bd6-70e8-443a-91fb-b07c84d3e9f9" containerID="7d650253985f12ab684dbab682cc5ef09f45adfc29e3f2e6afbd55eb539b4212" exitCode=0
Sep 09 20:54:04.305477 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:04.305146    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7643f-w28nz" event={"ID":"0dca9bd6-70e8-443a-91fb-b07c84d3e9f9","Type":"ContainerDied","Data":"7d650253985f12ab684dbab682cc5ef09f45adfc29e3f2e6afbd55eb539b4212"}
Sep 09 20:54:04.305477 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:04.305184    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7643f-w28nz" event={"ID":"0dca9bd6-70e8-443a-91fb-b07c84d3e9f9","Type":"ContainerStarted","Data":"5622894d4d8dee80bac9492e8ba8577ec438144df50b0d8d10494c2f6d6f5435"}
Sep 09 20:54:05.445378 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:05.445353    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7643f-w28nz"
Sep 09 20:54:05.542852 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:05.542829    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dca9bd6-70e8-443a-91fb-b07c84d3e9f9-maas-api-service-ca\") pod \"0dca9bd6-70e8-443a-91fb-b07c84d3e9f9\" (UID: \"0dca9bd6-70e8-443a-91fb-b07c84d3e9f9\") "
Sep 09 20:54:05.543187 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:05.543163    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dca9bd6-70e8-443a-91fb-b07c84d3e9f9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0dca9bd6-70e8-443a-91fb-b07c84d3e9f9" (UID: "0dca9bd6-70e8-443a-91fb-b07c84d3e9f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:05.643809 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:05.643752    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dca9bd6-70e8-443a-91fb-b07c84d3e9f9-maas-api-service-ca\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:06.317044 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:06.317011    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7643f-w28nz"
Sep 09 20:54:06.317202 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:06.317051    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7643f-w28nz" event={"ID":"0dca9bd6-70e8-443a-91fb-b07c84d3e9f9","Type":"ContainerDied","Data":"5622894d4d8dee80bac9492e8ba8577ec438144df50b0d8d10494c2f6d6f5435"}
Sep 09 20:54:06.317202 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:06.317096    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5622894d4d8dee80bac9492e8ba8577ec438144df50b0d8d10494c2f6d6f5435"
Sep 09 20:54:13.841953 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:13.841897    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7643f-w28nz"]
Sep 09 20:54:13.844077 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:13.844054    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7643f-w28nz"]
Sep 09 20:54:14.893720 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:14.893681    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dca9bd6-70e8-443a-91fb-b07c84d3e9f9" path="/var/lib/kubelet/pods/0dca9bd6-70e8-443a-91fb-b07c84d3e9f9/volumes"
Sep 09 20:54:22.178234 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:22.178196    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"]
Sep 09 20:54:22.827433 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:22.827394    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"]
Sep 09 20:54:30.729695 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:30.729661    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:54:31.246104 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.246074    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"]
Sep 09 20:54:31.246354 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.246268    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn" podUID="59076416-48b5-44bf-93cd-15e75a5b6e60" containerName="maas-api" containerID="cri-o://273bff863f6db0a2a0e6050c26874d209a6bdc243358664d49a39897bda20b97" gracePeriod=30
Sep 09 20:54:31.294585 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.294554    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l"]
Sep 09 20:54:31.294833 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.294806    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l" podUID="63b0cd0a-cf56-4821-915a-117f5b75556e" containerName="payload-processing" containerID="cri-o://55ab9816c069b0c6002549090fb7cde312658e49c584a01573c93f15fbd32b76" gracePeriod=30
Sep 09 20:54:31.304041 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.303999    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx"]
Sep 09 20:54:31.305206 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.305148    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx" podUID="997cb48e-b246-4ae0-b2d2-86fe8fe7ef00" containerName="payload-pre-processing" containerID="cri-o://1b720c7bd9c6966301a16834ab2b5e3fb5b0805a512483cf70c18bca2c8dbd92" gracePeriod=30
Sep 09 20:54:31.455427 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.455371    2556 generic.go:356] "Generic (PLEG): container finished" podID="59076416-48b5-44bf-93cd-15e75a5b6e60" containerID="273bff863f6db0a2a0e6050c26874d209a6bdc243358664d49a39897bda20b97" exitCode=0
Sep 09 20:54:31.455636 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.455440    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn" event={"ID":"59076416-48b5-44bf-93cd-15e75a5b6e60","Type":"ContainerDied","Data":"273bff863f6db0a2a0e6050c26874d209a6bdc243358664d49a39897bda20b97"}
Sep 09 20:54:31.457573 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.457364    2556 generic.go:356] "Generic (PLEG): container finished" podID="63b0cd0a-cf56-4821-915a-117f5b75556e" containerID="55ab9816c069b0c6002549090fb7cde312658e49c584a01573c93f15fbd32b76" exitCode=0
Sep 09 20:54:31.457573 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.457505    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l" event={"ID":"63b0cd0a-cf56-4821-915a-117f5b75556e","Type":"ContainerDied","Data":"55ab9816c069b0c6002549090fb7cde312658e49c584a01573c93f15fbd32b76"}
Sep 09 20:54:31.459811 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.459663    2556 generic.go:356] "Generic (PLEG): container finished" podID="997cb48e-b246-4ae0-b2d2-86fe8fe7ef00" containerID="1b720c7bd9c6966301a16834ab2b5e3fb5b0805a512483cf70c18bca2c8dbd92" exitCode=0
Sep 09 20:54:31.459811 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.459765    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx" event={"ID":"997cb48e-b246-4ae0-b2d2-86fe8fe7ef00","Type":"ContainerDied","Data":"1b720c7bd9c6966301a16834ab2b5e3fb5b0805a512483cf70c18bca2c8dbd92"}
Sep 09 20:54:31.585006 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.584962    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"
Sep 09 20:54:31.609985 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.609888    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l"
Sep 09 20:54:31.615084 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.615057    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx"
Sep 09 20:54:31.682732 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.682706    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59076416-48b5-44bf-93cd-15e75a5b6e60-metrics-certs\") pod \"59076416-48b5-44bf-93cd-15e75a5b6e60\" (UID: \"59076416-48b5-44bf-93cd-15e75a5b6e60\") "
Sep 09 20:54:31.682899 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.682758    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htlrk\" (UniqueName: \"kubernetes.io/projected/63b0cd0a-cf56-4821-915a-117f5b75556e-kube-api-access-htlrk\") pod \"63b0cd0a-cf56-4821-915a-117f5b75556e\" (UID: \"63b0cd0a-cf56-4821-915a-117f5b75556e\") "
Sep 09 20:54:31.682899 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.682779    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsbpf\" (UniqueName: \"kubernetes.io/projected/59076416-48b5-44bf-93cd-15e75a5b6e60-kube-api-access-tsbpf\") pod \"59076416-48b5-44bf-93cd-15e75a5b6e60\" (UID: \"59076416-48b5-44bf-93cd-15e75a5b6e60\") "
Sep 09 20:54:31.683082 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.683000    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhbjc\" (UniqueName: \"kubernetes.io/projected/997cb48e-b246-4ae0-b2d2-86fe8fe7ef00-kube-api-access-jhbjc\") pod \"997cb48e-b246-4ae0-b2d2-86fe8fe7ef00\" (UID: \"997cb48e-b246-4ae0-b2d2-86fe8fe7ef00\") "
Sep 09 20:54:31.683082 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.683073    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59076416-48b5-44bf-93cd-15e75a5b6e60-maas-api-tls\") pod \"59076416-48b5-44bf-93cd-15e75a5b6e60\" (UID: \"59076416-48b5-44bf-93cd-15e75a5b6e60\") "
Sep 09 20:54:31.683187 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.683134    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b0cd0a-cf56-4821-915a-117f5b75556e-plugins-config-volume\") pod \"63b0cd0a-cf56-4821-915a-117f5b75556e\" (UID: \"63b0cd0a-cf56-4821-915a-117f5b75556e\") "
Sep 09 20:54:31.683187 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.683164    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59076416-48b5-44bf-93cd-15e75a5b6e60-openshift-service-ca\") pod \"59076416-48b5-44bf-93cd-15e75a5b6e60\" (UID: \"59076416-48b5-44bf-93cd-15e75a5b6e60\") "
Sep 09 20:54:31.683187 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.683186    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/997cb48e-b246-4ae0-b2d2-86fe8fe7ef00-plugins-config-volume\") pod \"997cb48e-b246-4ae0-b2d2-86fe8fe7ef00\" (UID: \"997cb48e-b246-4ae0-b2d2-86fe8fe7ef00\") "
Sep 09 20:54:31.683579 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.683545    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b0cd0a-cf56-4821-915a-117f5b75556e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "63b0cd0a-cf56-4821-915a-117f5b75556e" (UID: "63b0cd0a-cf56-4821-915a-117f5b75556e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:31.683579 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.683567    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997cb48e-b246-4ae0-b2d2-86fe8fe7ef00-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "997cb48e-b246-4ae0-b2d2-86fe8fe7ef00" (UID: "997cb48e-b246-4ae0-b2d2-86fe8fe7ef00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:31.683850 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.683827    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/997cb48e-b246-4ae0-b2d2-86fe8fe7ef00-plugins-config-volume\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:31.683947 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.683858    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b0cd0a-cf56-4821-915a-117f5b75556e-plugins-config-volume\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:31.683947 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.683883    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59076416-48b5-44bf-93cd-15e75a5b6e60-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "59076416-48b5-44bf-93cd-15e75a5b6e60" (UID: "59076416-48b5-44bf-93cd-15e75a5b6e60"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:31.685287 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.685263    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59076416-48b5-44bf-93cd-15e75a5b6e60-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "59076416-48b5-44bf-93cd-15e75a5b6e60" (UID: "59076416-48b5-44bf-93cd-15e75a5b6e60"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:31.685506 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.685486    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b0cd0a-cf56-4821-915a-117f5b75556e-kube-api-access-htlrk" (OuterVolumeSpecName: "kube-api-access-htlrk") pod "63b0cd0a-cf56-4821-915a-117f5b75556e" (UID: "63b0cd0a-cf56-4821-915a-117f5b75556e"). InnerVolumeSpecName "kube-api-access-htlrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:31.685579 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.685506    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59076416-48b5-44bf-93cd-15e75a5b6e60-kube-api-access-tsbpf" (OuterVolumeSpecName: "kube-api-access-tsbpf") pod "59076416-48b5-44bf-93cd-15e75a5b6e60" (UID: "59076416-48b5-44bf-93cd-15e75a5b6e60"). InnerVolumeSpecName "kube-api-access-tsbpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:31.685579 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.685514    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997cb48e-b246-4ae0-b2d2-86fe8fe7ef00-kube-api-access-jhbjc" (OuterVolumeSpecName: "kube-api-access-jhbjc") pod "997cb48e-b246-4ae0-b2d2-86fe8fe7ef00" (UID: "997cb48e-b246-4ae0-b2d2-86fe8fe7ef00"). InnerVolumeSpecName "kube-api-access-jhbjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:31.685751 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.685737    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59076416-48b5-44bf-93cd-15e75a5b6e60-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "59076416-48b5-44bf-93cd-15e75a5b6e60" (UID: "59076416-48b5-44bf-93cd-15e75a5b6e60"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:31.784697 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.784629    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59076416-48b5-44bf-93cd-15e75a5b6e60-openshift-service-ca\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:31.784697 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.784656    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59076416-48b5-44bf-93cd-15e75a5b6e60-metrics-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:31.784697 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.784669    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htlrk\" (UniqueName: \"kubernetes.io/projected/63b0cd0a-cf56-4821-915a-117f5b75556e-kube-api-access-htlrk\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:31.784697 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.784681    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsbpf\" (UniqueName: \"kubernetes.io/projected/59076416-48b5-44bf-93cd-15e75a5b6e60-kube-api-access-tsbpf\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:31.784697 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.784695    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhbjc\" (UniqueName: \"kubernetes.io/projected/997cb48e-b246-4ae0-b2d2-86fe8fe7ef00-kube-api-access-jhbjc\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:31.785132 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:31.784708    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59076416-48b5-44bf-93cd-15e75a5b6e60-maas-api-tls\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:32.466867 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:32.466839    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx"
Sep 09 20:54:32.466867 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:32.466844    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx" event={"ID":"997cb48e-b246-4ae0-b2d2-86fe8fe7ef00","Type":"ContainerDied","Data":"6baeddbe136bd65d9bcf94aefe0c7cd47afcdb327546426c6a344b32eaed9829"}
Sep 09 20:54:32.467094 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:32.466940    2556 scope.go:117] "RemoveContainer" containerID="1b720c7bd9c6966301a16834ab2b5e3fb5b0805a512483cf70c18bca2c8dbd92"
Sep 09 20:54:32.468436 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:32.468397    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"
Sep 09 20:54:32.468570 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:32.468398    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn" event={"ID":"59076416-48b5-44bf-93cd-15e75a5b6e60","Type":"ContainerDied","Data":"999798e97bef21d47d14cf61be37c3c3f5ad2f765c8db6b5c0e71356976bf4f1"}
Sep 09 20:54:32.470045 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:32.470015    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l" event={"ID":"63b0cd0a-cf56-4821-915a-117f5b75556e","Type":"ContainerDied","Data":"73ce39c04c95a550bfc59494bbc8daf0563ef26b55c36d70c927e5f0aa15b35e"}
Sep 09 20:54:32.470045 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:32.470038    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l"
Sep 09 20:54:32.481349 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:32.479100    2556 scope.go:117] "RemoveContainer" containerID="273bff863f6db0a2a0e6050c26874d209a6bdc243358664d49a39897bda20b97"
Sep 09 20:54:32.490395 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:32.490270    2556 scope.go:117] "RemoveContainer" containerID="55ab9816c069b0c6002549090fb7cde312658e49c584a01573c93f15fbd32b76"
Sep 09 20:54:32.492766 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:32.492744    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx"]
Sep 09 20:54:32.497158 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:32.497138    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2dd30-76b6d4fdcd-5bjtx"]
Sep 09 20:54:32.508507 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:32.508487    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l"]
Sep 09 20:54:32.512631 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:32.512586    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2dd30-9585474c7-6bc6l"]
Sep 09 20:54:32.526654 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:32.526634    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"]
Sep 09 20:54:32.531755 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:32.531737    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b2dd30-68f54d84d9-lg9dn"]
Sep 09 20:54:32.893392 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:32.893311    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59076416-48b5-44bf-93cd-15e75a5b6e60" path="/var/lib/kubelet/pods/59076416-48b5-44bf-93cd-15e75a5b6e60/volumes"
Sep 09 20:54:32.893740 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:32.893667    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63b0cd0a-cf56-4821-915a-117f5b75556e" path="/var/lib/kubelet/pods/63b0cd0a-cf56-4821-915a-117f5b75556e/volumes"
Sep 09 20:54:32.894022 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:32.894008    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="997cb48e-b246-4ae0-b2d2-86fe8fe7ef00" path="/var/lib/kubelet/pods/997cb48e-b246-4ae0-b2d2-86fe8fe7ef00/volumes"
Sep 09 20:54:37.198384 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.198342    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q" podUID="65e66050-c303-4cfa-88ad-6bad1b65eb0b" containerName="main" containerID="cri-o://be2fe42661b7e52d9792767d1eeaa264771cb4a11bbb1649996146abd2a0eafb" gracePeriod=45
Sep 09 20:54:37.449434 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.449382    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:54:37.501174 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.501135    2556 generic.go:356] "Generic (PLEG): container finished" podID="65e66050-c303-4cfa-88ad-6bad1b65eb0b" containerID="be2fe42661b7e52d9792767d1eeaa264771cb4a11bbb1649996146abd2a0eafb" exitCode=0
Sep 09 20:54:37.501325 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.501181    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"
Sep 09 20:54:37.501325 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.501218    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q" event={"ID":"65e66050-c303-4cfa-88ad-6bad1b65eb0b","Type":"ContainerDied","Data":"be2fe42661b7e52d9792767d1eeaa264771cb4a11bbb1649996146abd2a0eafb"}
Sep 09 20:54:37.501325 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.501266    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q" event={"ID":"65e66050-c303-4cfa-88ad-6bad1b65eb0b","Type":"ContainerDied","Data":"ddb8045fc61e1d010cf7cc673b6863995792ed3f7e612cbdc1a363ef1cfdd60f"}
Sep 09 20:54:37.501325 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.501284    2556 scope.go:117] "RemoveContainer" containerID="be2fe42661b7e52d9792767d1eeaa264771cb4a11bbb1649996146abd2a0eafb"
Sep 09 20:54:37.509804 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.509788    2556 scope.go:117] "RemoveContainer" containerID="be2fe42661b7e52d9792767d1eeaa264771cb4a11bbb1649996146abd2a0eafb"
Sep 09 20:54:37.510076 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:54:37.510059    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be2fe42661b7e52d9792767d1eeaa264771cb4a11bbb1649996146abd2a0eafb\": container with ID starting with be2fe42661b7e52d9792767d1eeaa264771cb4a11bbb1649996146abd2a0eafb not found: ID does not exist" containerID="be2fe42661b7e52d9792767d1eeaa264771cb4a11bbb1649996146abd2a0eafb"
Sep 09 20:54:37.510131 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.510084    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be2fe42661b7e52d9792767d1eeaa264771cb4a11bbb1649996146abd2a0eafb"} err="failed to get container status \"be2fe42661b7e52d9792767d1eeaa264771cb4a11bbb1649996146abd2a0eafb\": rpc error: code = NotFound desc = could not find container \"be2fe42661b7e52d9792767d1eeaa264771cb4a11bbb1649996146abd2a0eafb\": container with ID starting with be2fe42661b7e52d9792767d1eeaa264771cb4a11bbb1649996146abd2a0eafb not found: ID does not exist"
Sep 09 20:54:37.537447 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.537428    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-model-cache\") pod \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") "
Sep 09 20:54:37.537520 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.537455    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65e66050-c303-4cfa-88ad-6bad1b65eb0b-tls-certs\") pod \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") "
Sep 09 20:54:37.537520 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.537472    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-dshm\") pod \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") "
Sep 09 20:54:37.537520 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.537485    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-home\") pod \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") "
Sep 09 20:54:37.537651 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.537579    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-tmp-dir\") pod \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") "
Sep 09 20:54:37.537698 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.537669    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcc42\" (UniqueName: \"kubernetes.io/projected/65e66050-c303-4cfa-88ad-6bad1b65eb0b-kube-api-access-mcc42\") pod \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\" (UID: \"65e66050-c303-4cfa-88ad-6bad1b65eb0b\") "
Sep 09 20:54:37.537751 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.537705    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-model-cache" (OuterVolumeSpecName: "model-cache") pod "65e66050-c303-4cfa-88ad-6bad1b65eb0b" (UID: "65e66050-c303-4cfa-88ad-6bad1b65eb0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:37.537825 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.537799    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-home" (OuterVolumeSpecName: "home") pod "65e66050-c303-4cfa-88ad-6bad1b65eb0b" (UID: "65e66050-c303-4cfa-88ad-6bad1b65eb0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:37.537887 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.537820    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "65e66050-c303-4cfa-88ad-6bad1b65eb0b" (UID: "65e66050-c303-4cfa-88ad-6bad1b65eb0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:37.538112 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.538090    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-tmp-dir\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:37.538179 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.538116    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-model-cache\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:37.538179 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.538130    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-home\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:37.539557 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.539538    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65e66050-c303-4cfa-88ad-6bad1b65eb0b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "65e66050-c303-4cfa-88ad-6bad1b65eb0b" (UID: "65e66050-c303-4cfa-88ad-6bad1b65eb0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:37.539642 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.539580    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-dshm" (OuterVolumeSpecName: "dshm") pod "65e66050-c303-4cfa-88ad-6bad1b65eb0b" (UID: "65e66050-c303-4cfa-88ad-6bad1b65eb0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:37.539722 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.539704    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e66050-c303-4cfa-88ad-6bad1b65eb0b-kube-api-access-mcc42" (OuterVolumeSpecName: "kube-api-access-mcc42") pod "65e66050-c303-4cfa-88ad-6bad1b65eb0b" (UID: "65e66050-c303-4cfa-88ad-6bad1b65eb0b"). InnerVolumeSpecName "kube-api-access-mcc42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:37.639274 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.639240    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcc42\" (UniqueName: \"kubernetes.io/projected/65e66050-c303-4cfa-88ad-6bad1b65eb0b-kube-api-access-mcc42\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:37.639274 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.639272    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65e66050-c303-4cfa-88ad-6bad1b65eb0b-tls-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:37.639274 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.639283    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65e66050-c303-4cfa-88ad-6bad1b65eb0b-dshm\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:37.824735 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.824710    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"]
Sep 09 20:54:37.828698 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.828679    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b2dd30/test-model-b2dd30-kserve-f7b7db955-cgw6q"]
Sep 09 20:54:37.847732 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:37.847709    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm" podUID="5dcf6537-1246-4938-80c1-77cdbb93a66d" containerName="main" containerID="cri-o://dd68182313155d20d96f244a729d1c1e181e16580d7c0efe5f2562e7f5676e1a" gracePeriod=45
Sep 09 20:54:38.078886 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.078857    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:54:38.144051 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.144021    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-model-cache\") pod \"5dcf6537-1246-4938-80c1-77cdbb93a66d\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") "
Sep 09 20:54:38.144226 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.144071    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95blv\" (UniqueName: \"kubernetes.io/projected/5dcf6537-1246-4938-80c1-77cdbb93a66d-kube-api-access-95blv\") pod \"5dcf6537-1246-4938-80c1-77cdbb93a66d\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") "
Sep 09 20:54:38.144226 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.144131    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-dshm\") pod \"5dcf6537-1246-4938-80c1-77cdbb93a66d\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") "
Sep 09 20:54:38.144226 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.144154    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-home\") pod \"5dcf6537-1246-4938-80c1-77cdbb93a66d\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") "
Sep 09 20:54:38.144226 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.144199    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dcf6537-1246-4938-80c1-77cdbb93a66d-tls-certs\") pod \"5dcf6537-1246-4938-80c1-77cdbb93a66d\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") "
Sep 09 20:54:38.144431 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.144254    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-tmp-dir\") pod \"5dcf6537-1246-4938-80c1-77cdbb93a66d\" (UID: \"5dcf6537-1246-4938-80c1-77cdbb93a66d\") "
Sep 09 20:54:38.144431 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.144315    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-model-cache" (OuterVolumeSpecName: "model-cache") pod "5dcf6537-1246-4938-80c1-77cdbb93a66d" (UID: "5dcf6537-1246-4938-80c1-77cdbb93a66d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:38.144546 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.144475    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-home" (OuterVolumeSpecName: "home") pod "5dcf6537-1246-4938-80c1-77cdbb93a66d" (UID: "5dcf6537-1246-4938-80c1-77cdbb93a66d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:38.144624 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.144602    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5dcf6537-1246-4938-80c1-77cdbb93a66d" (UID: "5dcf6537-1246-4938-80c1-77cdbb93a66d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:38.144730 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.144715    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-tmp-dir\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:38.144781 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.144735    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-model-cache\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:38.144781 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.144747    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-home\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:38.146364 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.146338    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dcf6537-1246-4938-80c1-77cdbb93a66d-kube-api-access-95blv" (OuterVolumeSpecName: "kube-api-access-95blv") pod "5dcf6537-1246-4938-80c1-77cdbb93a66d" (UID: "5dcf6537-1246-4938-80c1-77cdbb93a66d"). InnerVolumeSpecName "kube-api-access-95blv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:38.146625 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.146603    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dcf6537-1246-4938-80c1-77cdbb93a66d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5dcf6537-1246-4938-80c1-77cdbb93a66d" (UID: "5dcf6537-1246-4938-80c1-77cdbb93a66d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:38.146709 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.146633    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-dshm" (OuterVolumeSpecName: "dshm") pod "5dcf6537-1246-4938-80c1-77cdbb93a66d" (UID: "5dcf6537-1246-4938-80c1-77cdbb93a66d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:38.245682 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.245659    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dcf6537-1246-4938-80c1-77cdbb93a66d-tls-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:38.245952 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.245684    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95blv\" (UniqueName: \"kubernetes.io/projected/5dcf6537-1246-4938-80c1-77cdbb93a66d-kube-api-access-95blv\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:38.245952 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.245695    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dcf6537-1246-4938-80c1-77cdbb93a66d-dshm\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:38.507232 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.507203    2556 generic.go:356] "Generic (PLEG): container finished" podID="5dcf6537-1246-4938-80c1-77cdbb93a66d" containerID="dd68182313155d20d96f244a729d1c1e181e16580d7c0efe5f2562e7f5676e1a" exitCode=0
Sep 09 20:54:38.507373 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.507243    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"
Sep 09 20:54:38.507373 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.507298    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm" event={"ID":"5dcf6537-1246-4938-80c1-77cdbb93a66d","Type":"ContainerDied","Data":"dd68182313155d20d96f244a729d1c1e181e16580d7c0efe5f2562e7f5676e1a"}
Sep 09 20:54:38.507373 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.507340    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm" event={"ID":"5dcf6537-1246-4938-80c1-77cdbb93a66d","Type":"ContainerDied","Data":"ccdbb2f47b6c787046ebb34c4367d5d9a333a877961e8b00c53f7dbaf42e6b60"}
Sep 09 20:54:38.507373 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.507362    2556 scope.go:117] "RemoveContainer" containerID="dd68182313155d20d96f244a729d1c1e181e16580d7c0efe5f2562e7f5676e1a"
Sep 09 20:54:38.516444 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.516430    2556 scope.go:117] "RemoveContainer" containerID="dd68182313155d20d96f244a729d1c1e181e16580d7c0efe5f2562e7f5676e1a"
Sep 09 20:54:38.516719 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:54:38.516699    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd68182313155d20d96f244a729d1c1e181e16580d7c0efe5f2562e7f5676e1a\": container with ID starting with dd68182313155d20d96f244a729d1c1e181e16580d7c0efe5f2562e7f5676e1a not found: ID does not exist" containerID="dd68182313155d20d96f244a729d1c1e181e16580d7c0efe5f2562e7f5676e1a"
Sep 09 20:54:38.516774 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.516728    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd68182313155d20d96f244a729d1c1e181e16580d7c0efe5f2562e7f5676e1a"} err="failed to get container status \"dd68182313155d20d96f244a729d1c1e181e16580d7c0efe5f2562e7f5676e1a\": rpc error: code = NotFound desc = could not find container \"dd68182313155d20d96f244a729d1c1e181e16580d7c0efe5f2562e7f5676e1a\": container with ID starting with dd68182313155d20d96f244a729d1c1e181e16580d7c0efe5f2562e7f5676e1a not found: ID does not exist"
Sep 09 20:54:38.529207 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.529185    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"]
Sep 09 20:54:38.535690 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.535670    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-a214cb/noten-model-a214cb-kserve-76dcbd6c5-7tmrm"]
Sep 09 20:54:38.893469 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.893403    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dcf6537-1246-4938-80c1-77cdbb93a66d" path="/var/lib/kubelet/pods/5dcf6537-1246-4938-80c1-77cdbb93a66d/volumes"
Sep 09 20:54:38.893792 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:38.893779    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65e66050-c303-4cfa-88ad-6bad1b65eb0b" path="/var/lib/kubelet/pods/65e66050-c303-4cfa-88ad-6bad1b65eb0b/volumes"
Sep 09 20:54:43.241826 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:43.241794    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"]
Sep 09 20:54:43.242239 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:43.242008    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz" podUID="fbbd858d-84a8-40c5-a9ea-036ee3597316" containerName="istio-proxy" containerID="cri-o://2c14522b69d13824c6c21b9e4e0deb530a15a72e57874f96cfda92591e18e083" gracePeriod=30
Sep 09 20:54:44.093018 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.092987    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"]
Sep 09 20:54:44.093533 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.093515    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63b0cd0a-cf56-4821-915a-117f5b75556e" containerName="payload-processing"
Sep 09 20:54:44.093533 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.093533    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="63b0cd0a-cf56-4821-915a-117f5b75556e" containerName="payload-processing"
Sep 09 20:54:44.093678 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.093549    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dcf6537-1246-4938-80c1-77cdbb93a66d" containerName="main"
Sep 09 20:54:44.093678 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.093555    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dcf6537-1246-4938-80c1-77cdbb93a66d" containerName="main"
Sep 09 20:54:44.093678 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.093567    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59076416-48b5-44bf-93cd-15e75a5b6e60" containerName="maas-api"
Sep 09 20:54:44.093678 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.093573    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="59076416-48b5-44bf-93cd-15e75a5b6e60" containerName="maas-api"
Sep 09 20:54:44.093678 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.093580    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="997cb48e-b246-4ae0-b2d2-86fe8fe7ef00" containerName="payload-pre-processing"
Sep 09 20:54:44.093678 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.093587    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="997cb48e-b246-4ae0-b2d2-86fe8fe7ef00" containerName="payload-pre-processing"
Sep 09 20:54:44.093678 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.093599    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dca9bd6-70e8-443a-91fb-b07c84d3e9f9" containerName="revoke-keys"
Sep 09 20:54:44.093678 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.093605    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dca9bd6-70e8-443a-91fb-b07c84d3e9f9" containerName="revoke-keys"
Sep 09 20:54:44.093678 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.093613    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65e66050-c303-4cfa-88ad-6bad1b65eb0b" containerName="main"
Sep 09 20:54:44.093678 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.093618    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="65e66050-c303-4cfa-88ad-6bad1b65eb0b" containerName="main"
Sep 09 20:54:44.094181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.093683    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dca9bd6-70e8-443a-91fb-b07c84d3e9f9" containerName="revoke-keys"
Sep 09 20:54:44.094181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.093693    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="997cb48e-b246-4ae0-b2d2-86fe8fe7ef00" containerName="payload-pre-processing"
Sep 09 20:54:44.094181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.093700    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="65e66050-c303-4cfa-88ad-6bad1b65eb0b" containerName="main"
Sep 09 20:54:44.094181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.093707    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="63b0cd0a-cf56-4821-915a-117f5b75556e" containerName="payload-processing"
Sep 09 20:54:44.094181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.093714    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dcf6537-1246-4938-80c1-77cdbb93a66d" containerName="main"
Sep 09 20:54:44.094181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.093722    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="59076416-48b5-44bf-93cd-15e75a5b6e60" containerName="maas-api"
Sep 09 20:54:44.098555 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.098532    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.101481 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.101462    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-cb82f2-openshift-default-dockercfg-7gh8d\""
Sep 09 20:54:44.111569 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.111545    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"]
Sep 09 20:54:44.200208 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.200175    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324975d3-3abe-433d-8f56-f75672e340ce-istio-token\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.200387 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.200229    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-workload-socket\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.200387 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.200274    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-workload-certs\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.200387 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.200351    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-istio-data\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.200572 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.200419    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz8pd\" (UniqueName: \"kubernetes.io/projected/324975d3-3abe-433d-8f56-f75672e340ce-kube-api-access-dz8pd\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.200572 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.200552    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324975d3-3abe-433d-8f56-f75672e340ce-istiod-ca-cert\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.200681 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.200598    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-credential-socket\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.200681 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.200624    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-istio-envoy\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.200681 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.200647    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324975d3-3abe-433d-8f56-f75672e340ce-istio-podinfo\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.302125 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.302096    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324975d3-3abe-433d-8f56-f75672e340ce-istiod-ca-cert\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.302522 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.302134    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-credential-socket\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.302522 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.302153    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-istio-envoy\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.302522 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.302169    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324975d3-3abe-433d-8f56-f75672e340ce-istio-podinfo\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.302522 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.302208    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324975d3-3abe-433d-8f56-f75672e340ce-istio-token\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.302522 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.302238    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-workload-socket\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.302522 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.302377    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-workload-certs\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.302522 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.302441    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-istio-data\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.302522 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.302514    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz8pd\" (UniqueName: \"kubernetes.io/projected/324975d3-3abe-433d-8f56-f75672e340ce-kube-api-access-dz8pd\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.302951 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.302529    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-credential-socket\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.302951 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.302600    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-workload-socket\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.302951 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.302747    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-workload-certs\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.302951 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.302842    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-istio-data\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.303144 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.303124    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324975d3-3abe-433d-8f56-f75672e340ce-istiod-ca-cert\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.305027 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.305004    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-istio-envoy\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.305206 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.305177    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324975d3-3abe-433d-8f56-f75672e340ce-istio-podinfo\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.311268 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.311246    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324975d3-3abe-433d-8f56-f75672e340ce-istio-token\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.311383 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.311363    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz8pd\" (UniqueName: \"kubernetes.io/projected/324975d3-3abe-433d-8f56-f75672e340ce-kube-api-access-dz8pd\") pod \"e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") " pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.411953 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.411892    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:44.698336 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.698306    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"]
Sep 09 20:54:44.701106 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:54:44.701075    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324975d3_3abe_433d_8f56_f75672e340ce.slice/crio-6bc978f7299373ca304dbd52498024803d2024b5f44e652a1c15a9d68439cb37 WatchSource:0}: Error finding container 6bc978f7299373ca304dbd52498024803d2024b5f44e652a1c15a9d68439cb37: Status 404 returned error can't find the container with id 6bc978f7299373ca304dbd52498024803d2024b5f44e652a1c15a9d68439cb37
Sep 09 20:54:44.703452 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.703416    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:54:44.703536 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.703481    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:54:44.703536 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:44.703509    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:54:45.562462 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:45.562424    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc" event={"ID":"324975d3-3abe-433d-8f56-f75672e340ce","Type":"ContainerStarted","Data":"d68894d83af93c791d7be1e46ed3d610f786bfa0d765c92c389a6c638c6306f0"}
Sep 09 20:54:45.562883 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:45.562488    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc" event={"ID":"324975d3-3abe-433d-8f56-f75672e340ce","Type":"ContainerStarted","Data":"6bc978f7299373ca304dbd52498024803d2024b5f44e652a1c15a9d68439cb37"}
Sep 09 20:54:45.586270 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:45.586212    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc" podStartSLOduration=1.5861909349999999 podStartE2EDuration="1.586190935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:45.584644407 +0000 UTC m=+1331.264604731" watchObservedRunningTime="2026-09-09 20:54:45.586190935 +0000 UTC m=+1331.266151246"
Sep 09 20:54:46.412337 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:46.412301    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:46.417317 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:46.417292    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:46.567353 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:46.567327    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:46.568090 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:46.568070    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:54:48.194282 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.194217    2556 patch_prober.go:28] interesting pod/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 20:54:48.194282 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.194280    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz" podUID="fbbd858d-84a8-40c5-a9ea-036ee3597316" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:54:48.493923 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.493878    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:54:48.579159 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.579132    2556 generic.go:356] "Generic (PLEG): container finished" podID="fbbd858d-84a8-40c5-a9ea-036ee3597316" containerID="2c14522b69d13824c6c21b9e4e0deb530a15a72e57874f96cfda92591e18e083" exitCode=0
Sep 09 20:54:48.579289 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.579179    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"
Sep 09 20:54:48.579289 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.579213    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz" event={"ID":"fbbd858d-84a8-40c5-a9ea-036ee3597316","Type":"ContainerDied","Data":"2c14522b69d13824c6c21b9e4e0deb530a15a72e57874f96cfda92591e18e083"}
Sep 09 20:54:48.579289 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.579265    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz" event={"ID":"fbbd858d-84a8-40c5-a9ea-036ee3597316","Type":"ContainerDied","Data":"d5a3d8ba640391fa1d0f4c39389a27127c8435b67830d2f20d85377b0f16a0fb"}
Sep 09 20:54:48.579289 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.579287    2556 scope.go:117] "RemoveContainer" containerID="2c14522b69d13824c6c21b9e4e0deb530a15a72e57874f96cfda92591e18e083"
Sep 09 20:54:48.588134 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.588119    2556 scope.go:117] "RemoveContainer" containerID="8083f01b6fa208c0eb6e67037b8ad02e2cfcc6d0cd3daafc03f01df602776080"
Sep 09 20:54:48.596124 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.596107    2556 scope.go:117] "RemoveContainer" containerID="2c14522b69d13824c6c21b9e4e0deb530a15a72e57874f96cfda92591e18e083"
Sep 09 20:54:48.596378 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:54:48.596361    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c14522b69d13824c6c21b9e4e0deb530a15a72e57874f96cfda92591e18e083\": container with ID starting with 2c14522b69d13824c6c21b9e4e0deb530a15a72e57874f96cfda92591e18e083 not found: ID does not exist" containerID="2c14522b69d13824c6c21b9e4e0deb530a15a72e57874f96cfda92591e18e083"
Sep 09 20:54:48.596425 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.596385    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c14522b69d13824c6c21b9e4e0deb530a15a72e57874f96cfda92591e18e083"} err="failed to get container status \"2c14522b69d13824c6c21b9e4e0deb530a15a72e57874f96cfda92591e18e083\": rpc error: code = NotFound desc = could not find container \"2c14522b69d13824c6c21b9e4e0deb530a15a72e57874f96cfda92591e18e083\": container with ID starting with 2c14522b69d13824c6c21b9e4e0deb530a15a72e57874f96cfda92591e18e083 not found: ID does not exist"
Sep 09 20:54:48.596425 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.596399    2556 scope.go:117] "RemoveContainer" containerID="8083f01b6fa208c0eb6e67037b8ad02e2cfcc6d0cd3daafc03f01df602776080"
Sep 09 20:54:48.596655 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:54:48.596637    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8083f01b6fa208c0eb6e67037b8ad02e2cfcc6d0cd3daafc03f01df602776080\": container with ID starting with 8083f01b6fa208c0eb6e67037b8ad02e2cfcc6d0cd3daafc03f01df602776080 not found: ID does not exist" containerID="8083f01b6fa208c0eb6e67037b8ad02e2cfcc6d0cd3daafc03f01df602776080"
Sep 09 20:54:48.596696 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.596662    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8083f01b6fa208c0eb6e67037b8ad02e2cfcc6d0cd3daafc03f01df602776080"} err="failed to get container status \"8083f01b6fa208c0eb6e67037b8ad02e2cfcc6d0cd3daafc03f01df602776080\": rpc error: code = NotFound desc = could not find container \"8083f01b6fa208c0eb6e67037b8ad02e2cfcc6d0cd3daafc03f01df602776080\": container with ID starting with 8083f01b6fa208c0eb6e67037b8ad02e2cfcc6d0cd3daafc03f01df602776080 not found: ID does not exist"
Sep 09 20:54:48.652021 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.652000    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-token\") pod \"fbbd858d-84a8-40c5-a9ea-036ee3597316\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") "
Sep 09 20:54:48.652119 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.652053    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-data\") pod \"fbbd858d-84a8-40c5-a9ea-036ee3597316\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") "
Sep 09 20:54:48.652119 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.652080    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56sck\" (UniqueName: \"kubernetes.io/projected/fbbd858d-84a8-40c5-a9ea-036ee3597316-kube-api-access-56sck\") pod \"fbbd858d-84a8-40c5-a9ea-036ee3597316\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") "
Sep 09 20:54:48.652208 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.652153    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-workload-certs\") pod \"fbbd858d-84a8-40c5-a9ea-036ee3597316\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") "
Sep 09 20:54:48.652208 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.652184    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-workload-socket\") pod \"fbbd858d-84a8-40c5-a9ea-036ee3597316\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") "
Sep 09 20:54:48.652288 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.652245    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-credential-socket\") pod \"fbbd858d-84a8-40c5-a9ea-036ee3597316\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") "
Sep 09 20:54:48.652288 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.652267    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-data" (OuterVolumeSpecName: "istio-data") pod "fbbd858d-84a8-40c5-a9ea-036ee3597316" (UID: "fbbd858d-84a8-40c5-a9ea-036ee3597316"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:48.652369 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.652355    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-envoy\") pod \"fbbd858d-84a8-40c5-a9ea-036ee3597316\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") "
Sep 09 20:54:48.652423 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.652394    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbbd858d-84a8-40c5-a9ea-036ee3597316-istiod-ca-cert\") pod \"fbbd858d-84a8-40c5-a9ea-036ee3597316\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") "
Sep 09 20:54:48.652423 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.652418    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-podinfo\") pod \"fbbd858d-84a8-40c5-a9ea-036ee3597316\" (UID: \"fbbd858d-84a8-40c5-a9ea-036ee3597316\") "
Sep 09 20:54:48.652639 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.652411    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fbbd858d-84a8-40c5-a9ea-036ee3597316" (UID: "fbbd858d-84a8-40c5-a9ea-036ee3597316"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:48.652732 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.652445    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fbbd858d-84a8-40c5-a9ea-036ee3597316" (UID: "fbbd858d-84a8-40c5-a9ea-036ee3597316"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:48.652732 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.652588    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fbbd858d-84a8-40c5-a9ea-036ee3597316" (UID: "fbbd858d-84a8-40c5-a9ea-036ee3597316"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:48.652832 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.652745    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbbd858d-84a8-40c5-a9ea-036ee3597316-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fbbd858d-84a8-40c5-a9ea-036ee3597316" (UID: "fbbd858d-84a8-40c5-a9ea-036ee3597316"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:48.652889 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.652870    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbbd858d-84a8-40c5-a9ea-036ee3597316-istiod-ca-cert\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:48.652979 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.652892    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-data\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:48.652979 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.652920    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-workload-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:48.652979 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.652935    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-workload-socket\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:48.652979 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.652948    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-credential-socket\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:48.654181 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.654162    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-token" (OuterVolumeSpecName: "istio-token") pod "fbbd858d-84a8-40c5-a9ea-036ee3597316" (UID: "fbbd858d-84a8-40c5-a9ea-036ee3597316"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:48.654378 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.654362    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fbbd858d-84a8-40c5-a9ea-036ee3597316" (UID: "fbbd858d-84a8-40c5-a9ea-036ee3597316"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 20:54:48.654614 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.654593    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fbbd858d-84a8-40c5-a9ea-036ee3597316" (UID: "fbbd858d-84a8-40c5-a9ea-036ee3597316"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:48.654661 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.654622    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbbd858d-84a8-40c5-a9ea-036ee3597316-kube-api-access-56sck" (OuterVolumeSpecName: "kube-api-access-56sck") pod "fbbd858d-84a8-40c5-a9ea-036ee3597316" (UID: "fbbd858d-84a8-40c5-a9ea-036ee3597316"). InnerVolumeSpecName "kube-api-access-56sck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:48.754075 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.754044    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-envoy\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:48.754175 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.754077    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-podinfo\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:48.754175 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.754092    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbbd858d-84a8-40c5-a9ea-036ee3597316-istio-token\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:48.754175 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.754107    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56sck\" (UniqueName: \"kubernetes.io/projected/fbbd858d-84a8-40c5-a9ea-036ee3597316-kube-api-access-56sck\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:54:48.903521 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.903496    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"]
Sep 09 20:54:48.912318 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:48.912294    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b2dd30-openshift-default-7cfbdf9f66-dvmbz"]
Sep 09 20:54:49.879366 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:49.879332    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj"]
Sep 09 20:54:49.879808 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:49.879793    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbbd858d-84a8-40c5-a9ea-036ee3597316" containerName="istio-proxy"
Sep 09 20:54:49.879862 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:49.879810    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbbd858d-84a8-40c5-a9ea-036ee3597316" containerName="istio-proxy"
Sep 09 20:54:49.879862 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:49.879824    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbbd858d-84a8-40c5-a9ea-036ee3597316" containerName="istio-proxy"
Sep 09 20:54:49.879862 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:49.879829    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbbd858d-84a8-40c5-a9ea-036ee3597316" containerName="istio-proxy"
Sep 09 20:54:49.879998 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:49.879893    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbbd858d-84a8-40c5-a9ea-036ee3597316" containerName="istio-proxy"
Sep 09 20:54:49.884635 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:49.884619    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj"
Sep 09 20:54:49.887591 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:49.887569    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-cb82f2\""
Sep 09 20:54:49.900740 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:49.900720    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj"]
Sep 09 20:54:49.966813 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:49.966783    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj\" (UID: \"2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj"
Sep 09 20:54:49.966932 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:49.966848    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqzbz\" (UniqueName: \"kubernetes.io/projected/2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3-kube-api-access-rqzbz\") pod \"payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj\" (UID: \"2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj"
Sep 09 20:54:50.067780 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:50.067745    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqzbz\" (UniqueName: \"kubernetes.io/projected/2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3-kube-api-access-rqzbz\") pod \"payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj\" (UID: \"2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj"
Sep 09 20:54:50.067974 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:50.067887    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj\" (UID: \"2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj"
Sep 09 20:54:50.068623 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:50.068600    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj\" (UID: \"2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj"
Sep 09 20:54:50.075856 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:50.075835    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqzbz\" (UniqueName: \"kubernetes.io/projected/2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3-kube-api-access-rqzbz\") pod \"payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj\" (UID: \"2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj"
Sep 09 20:54:50.195132 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:50.195066    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj"
Sep 09 20:54:50.468278 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:50.468246    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj"]
Sep 09 20:54:50.469455 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:54:50.469431    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6d8aeb_fbdd_47c5_87a5_fbcff33ec7e3.slice/crio-3aaf7a53d740ec556a40d72183ed9b0b7af2b1891c702555e233fbe1d4c5b253 WatchSource:0}: Error finding container 3aaf7a53d740ec556a40d72183ed9b0b7af2b1891c702555e233fbe1d4c5b253: Status 404 returned error can't find the container with id 3aaf7a53d740ec556a40d72183ed9b0b7af2b1891c702555e233fbe1d4c5b253
Sep 09 20:54:50.593362 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:50.593329    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj" event={"ID":"2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3","Type":"ContainerStarted","Data":"d7acb69071af33511e31df7c561e5f8d514a5e658965aab1a77758ae28eb976c"}
Sep 09 20:54:50.593506 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:50.593383    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj"
Sep 09 20:54:50.593506 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:50.593405    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj" event={"ID":"2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3","Type":"ContainerStarted","Data":"3aaf7a53d740ec556a40d72183ed9b0b7af2b1891c702555e233fbe1d4c5b253"}
Sep 09 20:54:50.612598 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:50.612557    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj" podStartSLOduration=1.612542404 podStartE2EDuration="1.612542404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:50.61168513 +0000 UTC m=+1336.291645439" watchObservedRunningTime="2026-09-09 20:54:50.612542404 +0000 UTC m=+1336.292502712"
Sep 09 20:54:50.894488 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:54:50.894450    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbbd858d-84a8-40c5-a9ea-036ee3597316" path="/var/lib/kubelet/pods/fbbd858d-84a8-40c5-a9ea-036ee3597316/volumes"
Sep 09 20:55:01.601269 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:01.601184    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj"
Sep 09 20:55:09.012026 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:09.011993    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:55:13.100934 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.100065    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"]
Sep 09 20:55:13.100934 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.100841    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbbd858d-84a8-40c5-a9ea-036ee3597316" containerName="istio-proxy"
Sep 09 20:55:13.104722 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.104699    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.108861 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.108841    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-a2fb5a-openshift-default-dockercfg-ctjv6\""
Sep 09 20:55:13.123799 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.123772    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"]
Sep 09 20:55:13.185571 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.185532    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-credential-socket\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.185826 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.185805    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-podinfo\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.185994 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.185975    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-data\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.186125 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.186108    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-token\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.186267 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.186252    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-envoy\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.186392 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.186377    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-workload-certs\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.186526 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.186508    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-workload-socket\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.186634 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.186618    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfnrl\" (UniqueName: \"kubernetes.io/projected/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-kube-api-access-rfnrl\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.186758 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.186742    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istiod-ca-cert\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.287836 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.287798    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-workload-socket\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.288107 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.288083    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfnrl\" (UniqueName: \"kubernetes.io/projected/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-kube-api-access-rfnrl\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.288270 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.288252    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istiod-ca-cert\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.288465 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.288431    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-workload-socket\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.288563 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.288446    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-credential-socket\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.288694 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.288678    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-podinfo\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.288811 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.288797    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-data\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.288941 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.288923    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-token\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.289103 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.289088    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-envoy\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.289218 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.289197    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-credential-socket\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.289303 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.289289    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-workload-certs\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.290321 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.290291    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istiod-ca-cert\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.291169 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.291135    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-workload-certs\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.291289 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.291242    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-data\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.294556 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.294234    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-envoy\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.296528 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.296505    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-podinfo\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.307129 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.307111    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfnrl\" (UniqueName: \"kubernetes.io/projected/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-kube-api-access-rfnrl\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.312047 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.312025    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-token\") pod \"e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") " pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.416094 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.416010    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:13.701639 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.701438    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"]
Sep 09 20:55:13.703990 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:55:13.703961    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ecb41f5_9336_41eb_8fdc_9a5a8c6d991b.slice/crio-98d66a0d8b362b6b3c7747dfd578f3a7e28832072a46eaf81c7172f81eb0b0ab WatchSource:0}: Error finding container 98d66a0d8b362b6b3c7747dfd578f3a7e28832072a46eaf81c7172f81eb0b0ab: Status 404 returned error can't find the container with id 98d66a0d8b362b6b3c7747dfd578f3a7e28832072a46eaf81c7172f81eb0b0ab
Sep 09 20:55:13.706215 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.706183    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:55:13.706287 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.706252    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:55:13.706287 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.706281    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:55:13.723706 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:13.723680    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql" event={"ID":"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b","Type":"ContainerStarted","Data":"98d66a0d8b362b6b3c7747dfd578f3a7e28832072a46eaf81c7172f81eb0b0ab"}
Sep 09 20:55:14.730409 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:14.730378    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql" event={"ID":"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b","Type":"ContainerStarted","Data":"dd36d59c46513f4fa5e8bc1a47935def791714dd0320a0fdbc855a095dbdcbcc"}
Sep 09 20:55:14.761489 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:14.761439    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql" podStartSLOduration=1.761422859 podStartE2EDuration="1.761422859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:14.759046191 +0000 UTC m=+1360.439006511" watchObservedRunningTime="2026-09-09 20:55:14.761422859 +0000 UTC m=+1360.441383175"
Sep 09 20:55:15.416231 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:15.416199    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:15.421278 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:15.421252    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:15.736372 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:15.736343    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:15.737333 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:15.737314    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:17.610136 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:17.610089    2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 20:55:17.610616 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:17.610165    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc" podUID="324975d3-3abe-433d-8f56-f75672e340ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 20:55:19.761160 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:19.761128    2556 generic.go:356] "Generic (PLEG): container finished" podID="324975d3-3abe-433d-8f56-f75672e340ce" containerID="d68894d83af93c791d7be1e46ed3d610f786bfa0d765c92c389a6c638c6306f0" exitCode=0
Sep 09 20:55:19.761551 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:19.761181    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc" event={"ID":"324975d3-3abe-433d-8f56-f75672e340ce","Type":"ContainerDied","Data":"d68894d83af93c791d7be1e46ed3d610f786bfa0d765c92c389a6c638c6306f0"}
Sep 09 20:55:19.761621 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:19.761581    2556 scope.go:117] "RemoveContainer" containerID="d68894d83af93c791d7be1e46ed3d610f786bfa0d765c92c389a6c638c6306f0"
Sep 09 20:55:19.762738 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:19.762704    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:55:19.762814 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:19.762773    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:55:19.762814 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:19.762806    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:55:20.768278 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:20.768244    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc" event={"ID":"324975d3-3abe-433d-8f56-f75672e340ce","Type":"ContainerStarted","Data":"88d8523a1ee40d78d261f2cd7e2cbef6be0c637a3f886ee25d38a2a21f3389e5"}
Sep 09 20:55:21.412260 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:21.412216    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:55:21.412476 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:21.412442    2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 09 20:55:21.412525 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:21.412507    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc" podUID="324975d3-3abe-433d-8f56-f75672e340ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 09 20:55:22.412845 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:22.412801    2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 09 20:55:22.413272 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:22.412872    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc" podUID="324975d3-3abe-433d-8f56-f75672e340ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 09 20:55:23.413106 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:23.413070    2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 09 20:55:23.413525 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:23.413125    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc" podUID="324975d3-3abe-433d-8f56-f75672e340ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 09 20:55:24.413164 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:24.413128    2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 09 20:55:24.413545 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:24.413205    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc" podUID="324975d3-3abe-433d-8f56-f75672e340ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 09 20:55:25.412884 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:25.412847    2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 09 20:55:25.413058 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:25.412927    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc" podUID="324975d3-3abe-433d-8f56-f75672e340ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 09 20:55:26.413245 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:26.413205    2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 09 20:55:26.413635 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:26.413273    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc" podUID="324975d3-3abe-433d-8f56-f75672e340ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 09 20:55:27.416469 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:27.416425    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:55:27.416984 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:27.416769    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:55:27.417443 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:27.417421    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 20:55:44.001058 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:44.001027    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:55:55.968714 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:55.968682    2556 generic.go:356] "Generic (PLEG): container finished" podID="5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" containerID="dd36d59c46513f4fa5e8bc1a47935def791714dd0320a0fdbc855a095dbdcbcc" exitCode=0
Sep 09 20:55:55.969156 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:55.968764    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql" event={"ID":"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b","Type":"ContainerDied","Data":"dd36d59c46513f4fa5e8bc1a47935def791714dd0320a0fdbc855a095dbdcbcc"}
Sep 09 20:55:55.969156 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:55.969138    2556 scope.go:117] "RemoveContainer" containerID="dd36d59c46513f4fa5e8bc1a47935def791714dd0320a0fdbc855a095dbdcbcc"
Sep 09 20:55:55.970348 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:55.970322    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:55:55.970418 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:55.970388    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:55:55.970466 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:55.970418    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:55:56.979756 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:56.979720    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql" event={"ID":"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b","Type":"ContainerStarted","Data":"328380a79b50f1713af65f153b99c4ed7684579049c516a6ae2a85f0f669f61b"}
Sep 09 20:55:57.416466 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:57.416424    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:55:57.417728 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:57.417699    2556 patch_prober.go:28] interesting pod/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 09 20:55:57.417844 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:57.417761    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql" podUID="5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 09 20:55:58.417117 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:58.417077    2556 patch_prober.go:28] interesting pod/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 09 20:55:58.417510 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:58.417146    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql" podUID="5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 09 20:55:59.416767 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:59.416731    2556 patch_prober.go:28] interesting pod/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 09 20:55:59.416954 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:55:59.416785    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql" podUID="5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 09 20:56:00.417014 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:00.416981    2556 patch_prober.go:28] interesting pod/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 09 20:56:00.417411 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:00.417035    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql" podUID="5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 09 20:56:00.736237 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:00.736207    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:56:01.159516 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:01.159439    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fbe7be47-cm7vd"]
Sep 09 20:56:01.162155 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:01.162140    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fbe7be47-cm7vd"
Sep 09 20:56:01.164526 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:01.164504    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsks9\""
Sep 09 20:56:01.170137 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:01.170112    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fbe7be47-cm7vd"]
Sep 09 20:56:01.258654 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:01.258622    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b2c37cb-0639-4a21-87a2-1abcf7214f29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fbe7be47-cm7vd\" (UID: \"6b2c37cb-0639-4a21-87a2-1abcf7214f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fbe7be47-cm7vd"
Sep 09 20:56:01.359704 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:01.359669    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b2c37cb-0639-4a21-87a2-1abcf7214f29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fbe7be47-cm7vd\" (UID: \"6b2c37cb-0639-4a21-87a2-1abcf7214f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fbe7be47-cm7vd"
Sep 09 20:56:01.360301 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:01.360278    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b2c37cb-0639-4a21-87a2-1abcf7214f29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fbe7be47-cm7vd\" (UID: \"6b2c37cb-0639-4a21-87a2-1abcf7214f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fbe7be47-cm7vd"
Sep 09 20:56:01.417285 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:01.417202    2556 patch_prober.go:28] interesting pod/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 09 20:56:01.417285 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:01.417260    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql" podUID="5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 09 20:56:01.475542 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:01.475515    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fbe7be47-cm7vd"
Sep 09 20:56:01.761646 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:01.755971    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fbe7be47-cm7vd"]
Sep 09 20:56:02.013584 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:02.013509    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fbe7be47-cm7vd" event={"ID":"6b2c37cb-0639-4a21-87a2-1abcf7214f29","Type":"ContainerStarted","Data":"d622c0711d907fba8c7713299726efefe70d7e0400325735e31ac303785bf9d5"}
Sep 09 20:56:02.420804 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:02.420730    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:56:02.421632 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:02.421614    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 20:56:03.020517 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:03.020489    2556 generic.go:356] "Generic (PLEG): container finished" podID="6b2c37cb-0639-4a21-87a2-1abcf7214f29" containerID="75d4ced149c8c0c564aff588ba4d407c7048a175b4b7544b280a7a32ca9698ac" exitCode=0
Sep 09 20:56:03.020677 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:03.020572    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fbe7be47-cm7vd" event={"ID":"6b2c37cb-0639-4a21-87a2-1abcf7214f29","Type":"ContainerDied","Data":"75d4ced149c8c0c564aff588ba4d407c7048a175b4b7544b280a7a32ca9698ac"}
Sep 09 20:56:04.156679 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:04.156658    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fbe7be47-cm7vd"
Sep 09 20:56:04.287057 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:04.286993    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b2c37cb-0639-4a21-87a2-1abcf7214f29-maas-api-service-ca\") pod \"6b2c37cb-0639-4a21-87a2-1abcf7214f29\" (UID: \"6b2c37cb-0639-4a21-87a2-1abcf7214f29\") "
Sep 09 20:56:04.287378 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:04.287353    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2c37cb-0639-4a21-87a2-1abcf7214f29-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6b2c37cb-0639-4a21-87a2-1abcf7214f29" (UID: "6b2c37cb-0639-4a21-87a2-1abcf7214f29"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:56:04.287436 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:04.287421    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b2c37cb-0639-4a21-87a2-1abcf7214f29-maas-api-service-ca\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:56:05.033132 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:05.033102    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fbe7be47-cm7vd"
Sep 09 20:56:05.033289 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:05.033136    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fbe7be47-cm7vd" event={"ID":"6b2c37cb-0639-4a21-87a2-1abcf7214f29","Type":"ContainerDied","Data":"d622c0711d907fba8c7713299726efefe70d7e0400325735e31ac303785bf9d5"}
Sep 09 20:56:05.033289 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:05.033177    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d622c0711d907fba8c7713299726efefe70d7e0400325735e31ac303785bf9d5"
Sep 09 20:56:07.490347 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.490315    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"]
Sep 09 20:56:07.490969 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.490938    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b2c37cb-0639-4a21-87a2-1abcf7214f29" containerName="revoke-keys"
Sep 09 20:56:07.490969 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.490974    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b2c37cb-0639-4a21-87a2-1abcf7214f29" containerName="revoke-keys"
Sep 09 20:56:07.491071 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.491060    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b2c37cb-0639-4a21-87a2-1abcf7214f29" containerName="revoke-keys"
Sep 09 20:56:07.494165 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.494142    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:07.497787 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.497759    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-933e9d\"/\"unconfigured-w4-933e9d-kserve-self-signed-certs\""
Sep 09 20:56:07.503297 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.503276    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"]
Sep 09 20:56:07.615490 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.615460    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjb6q\" (UniqueName: \"kubernetes.io/projected/57b4976c-255a-4d22-884a-2492e4df333d-kube-api-access-mjb6q\") pod \"unconfigured-w4-933e9d-kserve-9dbd57777-g26cw\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") " pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:07.615646 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.615509    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-home\") pod \"unconfigured-w4-933e9d-kserve-9dbd57777-g26cw\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") " pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:07.615646 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.615541    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-tmp-dir\") pod \"unconfigured-w4-933e9d-kserve-9dbd57777-g26cw\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") " pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:07.615646 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.615590    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-dshm\") pod \"unconfigured-w4-933e9d-kserve-9dbd57777-g26cw\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") " pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:07.615646 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.615615    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-model-cache\") pod \"unconfigured-w4-933e9d-kserve-9dbd57777-g26cw\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") " pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:07.615861 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.615738    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57b4976c-255a-4d22-884a-2492e4df333d-tls-certs\") pod \"unconfigured-w4-933e9d-kserve-9dbd57777-g26cw\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") " pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:07.717323 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.717286    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57b4976c-255a-4d22-884a-2492e4df333d-tls-certs\") pod \"unconfigured-w4-933e9d-kserve-9dbd57777-g26cw\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") " pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:07.717511 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.717379    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjb6q\" (UniqueName: \"kubernetes.io/projected/57b4976c-255a-4d22-884a-2492e4df333d-kube-api-access-mjb6q\") pod \"unconfigured-w4-933e9d-kserve-9dbd57777-g26cw\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") " pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:07.717511 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.717419    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-home\") pod \"unconfigured-w4-933e9d-kserve-9dbd57777-g26cw\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") " pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:07.717511 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.717446    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-tmp-dir\") pod \"unconfigured-w4-933e9d-kserve-9dbd57777-g26cw\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") " pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:07.717511 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.717486    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-dshm\") pod \"unconfigured-w4-933e9d-kserve-9dbd57777-g26cw\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") " pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:07.717705 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.717512    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-model-cache\") pod \"unconfigured-w4-933e9d-kserve-9dbd57777-g26cw\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") " pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:07.717929 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.717883    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-home\") pod \"unconfigured-w4-933e9d-kserve-9dbd57777-g26cw\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") " pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:07.718035 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.717928    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-tmp-dir\") pod \"unconfigured-w4-933e9d-kserve-9dbd57777-g26cw\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") " pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:07.718035 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.717950    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-model-cache\") pod \"unconfigured-w4-933e9d-kserve-9dbd57777-g26cw\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") " pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:07.719624 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.719604    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-dshm\") pod \"unconfigured-w4-933e9d-kserve-9dbd57777-g26cw\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") " pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:07.719866 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.719850    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57b4976c-255a-4d22-884a-2492e4df333d-tls-certs\") pod \"unconfigured-w4-933e9d-kserve-9dbd57777-g26cw\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") " pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:07.726743 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.726723    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjb6q\" (UniqueName: \"kubernetes.io/projected/57b4976c-255a-4d22-884a-2492e4df333d-kube-api-access-mjb6q\") pod \"unconfigured-w4-933e9d-kserve-9dbd57777-g26cw\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") " pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:07.804467 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:07.804393    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:08.098689 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:08.098653    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"]
Sep 09 20:56:08.100012 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:56:08.099987    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57b4976c_255a_4d22_884a_2492e4df333d.slice/crio-873bbc5b52e41cbc00887d3ea4ffd949d7486a8e7ff79282e9c678a1a4d7daeb WatchSource:0}: Error finding container 873bbc5b52e41cbc00887d3ea4ffd949d7486a8e7ff79282e9c678a1a4d7daeb: Status 404 returned error can't find the container with id 873bbc5b52e41cbc00887d3ea4ffd949d7486a8e7ff79282e9c678a1a4d7daeb
Sep 09 20:56:08.156790 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:08.156764    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:56:09.056919 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:09.056882    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw" event={"ID":"57b4976c-255a-4d22-884a-2492e4df333d","Type":"ContainerStarted","Data":"d6c414d2e7b65807ac0c6d637966a6b073e5d3bf3775c965a54de1a7115c2874"}
Sep 09 20:56:09.057279 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:09.056941    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw" event={"ID":"57b4976c-255a-4d22-884a-2492e4df333d","Type":"ContainerStarted","Data":"873bbc5b52e41cbc00887d3ea4ffd949d7486a8e7ff79282e9c678a1a4d7daeb"}
Sep 09 20:56:09.081944 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:09.081881    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw" podStartSLOduration=2.081867086 podStartE2EDuration="2.081867086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:56:09.080991822 +0000 UTC m=+1414.760952132" watchObservedRunningTime="2026-09-09 20:56:09.081867086 +0000 UTC m=+1414.761827396"
Sep 09 20:56:13.286420 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.286388    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"]
Sep 09 20:56:13.290243 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.290218    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:13.292972 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.292950    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2b2343\"/\"unconfigured-w0-2b2343-kserve-self-signed-certs\""
Sep 09 20:56:13.300565 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.300542    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"]
Sep 09 20:56:13.377825 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.377797    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-model-cache\") pod \"unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") " pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:13.377957 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.377835    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-dshm\") pod \"unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") " pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:13.377957 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.377865    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea37b5f6-441f-458c-a4ba-6949c906cbfa-tls-certs\") pod \"unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") " pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:13.378051 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.377964    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-tmp-dir\") pod \"unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") " pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:13.378051 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.377990    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-home\") pod \"unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") " pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:13.378051 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.378046    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzpcc\" (UniqueName: \"kubernetes.io/projected/ea37b5f6-441f-458c-a4ba-6949c906cbfa-kube-api-access-tzpcc\") pod \"unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") " pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:13.478711 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.478679    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-tmp-dir\") pod \"unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") " pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:13.478883 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.478719    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-home\") pod \"unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") " pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:13.478883 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.478774    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzpcc\" (UniqueName: \"kubernetes.io/projected/ea37b5f6-441f-458c-a4ba-6949c906cbfa-kube-api-access-tzpcc\") pod \"unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") " pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:13.478883 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.478824    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-model-cache\") pod \"unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") " pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:13.478883 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.478852    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-dshm\") pod \"unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") " pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:13.478883 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.478883    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea37b5f6-441f-458c-a4ba-6949c906cbfa-tls-certs\") pod \"unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") " pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:13.479188 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.479156    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-tmp-dir\") pod \"unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") " pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:13.479240 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.479224    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-model-cache\") pod \"unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") " pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:13.479408 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.479390    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-home\") pod \"unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") " pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:13.481073 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.481043    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-dshm\") pod \"unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") " pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:13.481287 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.481271    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea37b5f6-441f-458c-a4ba-6949c906cbfa-tls-certs\") pod \"unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") " pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:13.489098 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.489079    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzpcc\" (UniqueName: \"kubernetes.io/projected/ea37b5f6-441f-458c-a4ba-6949c906cbfa-kube-api-access-tzpcc\") pod \"unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") " pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:13.600404 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.600352    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:13.873534 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:13.873511    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"]
Sep 09 20:56:13.876272 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:56:13.876240    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea37b5f6_441f_458c_a4ba_6949c906cbfa.slice/crio-02444f0b240ca0165c02031f6e97231afbc686662b1b4948009149b4eab519c2 WatchSource:0}: Error finding container 02444f0b240ca0165c02031f6e97231afbc686662b1b4948009149b4eab519c2: Status 404 returned error can't find the container with id 02444f0b240ca0165c02031f6e97231afbc686662b1b4948009149b4eab519c2
Sep 09 20:56:14.086732 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:14.086696    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8" event={"ID":"ea37b5f6-441f-458c-a4ba-6949c906cbfa","Type":"ContainerStarted","Data":"4e277ef37c992ab5fc21752270289d05df9e1cd967b42b0b75800977e575aa32"}
Sep 09 20:56:14.086884 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:14.086745    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8" event={"ID":"ea37b5f6-441f-458c-a4ba-6949c906cbfa","Type":"ContainerStarted","Data":"02444f0b240ca0165c02031f6e97231afbc686662b1b4948009149b4eab519c2"}
Sep 09 20:56:14.108771 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:14.108728    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8" podStartSLOduration=1.10871422 podStartE2EDuration="1.10871422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:56:14.106383059 +0000 UTC m=+1419.786343393" watchObservedRunningTime="2026-09-09 20:56:14.10871422 +0000 UTC m=+1419.788674520"
Sep 09 20:56:16.733988 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:16.733949    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf"]
Sep 09 20:56:16.734544 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:16.734194    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf" podUID="01483de9-50af-4cbc-8d3e-373e8a6fc1f8" containerName="payload-processing" containerID="cri-o://a34733430582e04c45975c5d5ad4912c8838a3f12ecdb4cd9289b6aa43ec2d75" gracePeriod=30
Sep 09 20:56:16.740049 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:16.740021    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l"]
Sep 09 20:56:16.740264 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:16.740232    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l" podUID="ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9" containerName="payload-pre-processing" containerID="cri-o://4fdf591f4c29c0a153f0fe318838d16a00afba7f8c39a4e646175d94109ef162" gracePeriod=30
Sep 09 20:56:16.935838 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:16.935803    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fbe7be47-cm7vd"]
Sep 09 20:56:16.938116 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:16.938087    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fbe7be47-cm7vd"]
Sep 09 20:56:16.993312 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:16.993261    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf"
Sep 09 20:56:17.011538 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.011516    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01483de9-50af-4cbc-8d3e-373e8a6fc1f8-plugins-config-volume\") pod \"01483de9-50af-4cbc-8d3e-373e8a6fc1f8\" (UID: \"01483de9-50af-4cbc-8d3e-373e8a6fc1f8\") "
Sep 09 20:56:17.011650 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.011630    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pg2k\" (UniqueName: \"kubernetes.io/projected/01483de9-50af-4cbc-8d3e-373e8a6fc1f8-kube-api-access-7pg2k\") pod \"01483de9-50af-4cbc-8d3e-373e8a6fc1f8\" (UID: \"01483de9-50af-4cbc-8d3e-373e8a6fc1f8\") "
Sep 09 20:56:17.011844 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.011821    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01483de9-50af-4cbc-8d3e-373e8a6fc1f8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "01483de9-50af-4cbc-8d3e-373e8a6fc1f8" (UID: "01483de9-50af-4cbc-8d3e-373e8a6fc1f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:56:17.012130 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.012114    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01483de9-50af-4cbc-8d3e-373e8a6fc1f8-plugins-config-volume\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:56:17.013622 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.013560    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01483de9-50af-4cbc-8d3e-373e8a6fc1f8-kube-api-access-7pg2k" (OuterVolumeSpecName: "kube-api-access-7pg2k") pod "01483de9-50af-4cbc-8d3e-373e8a6fc1f8" (UID: "01483de9-50af-4cbc-8d3e-373e8a6fc1f8"). InnerVolumeSpecName "kube-api-access-7pg2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:56:17.025640 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.025620    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l"
Sep 09 20:56:17.104920 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.104882    2556 generic.go:356] "Generic (PLEG): container finished" podID="01483de9-50af-4cbc-8d3e-373e8a6fc1f8" containerID="a34733430582e04c45975c5d5ad4912c8838a3f12ecdb4cd9289b6aa43ec2d75" exitCode=0
Sep 09 20:56:17.105120 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.104952    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf"
Sep 09 20:56:17.105120 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.104969    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf" event={"ID":"01483de9-50af-4cbc-8d3e-373e8a6fc1f8","Type":"ContainerDied","Data":"a34733430582e04c45975c5d5ad4912c8838a3f12ecdb4cd9289b6aa43ec2d75"}
Sep 09 20:56:17.105120 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.105028    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf" event={"ID":"01483de9-50af-4cbc-8d3e-373e8a6fc1f8","Type":"ContainerDied","Data":"c0fa7413c1e19411b73096c860803d9aead47fba6f7e76658b6aa2a09816f27d"}
Sep 09 20:56:17.105120 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.105061    2556 scope.go:117] "RemoveContainer" containerID="a34733430582e04c45975c5d5ad4912c8838a3f12ecdb4cd9289b6aa43ec2d75"
Sep 09 20:56:17.106961 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.106935    2556 generic.go:356] "Generic (PLEG): container finished" podID="ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9" containerID="4fdf591f4c29c0a153f0fe318838d16a00afba7f8c39a4e646175d94109ef162" exitCode=0
Sep 09 20:56:17.107075 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.107050    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l" event={"ID":"ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9","Type":"ContainerDied","Data":"4fdf591f4c29c0a153f0fe318838d16a00afba7f8c39a4e646175d94109ef162"}
Sep 09 20:56:17.107122 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.107068    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l"
Sep 09 20:56:17.107122 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.107094    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l" event={"ID":"ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9","Type":"ContainerDied","Data":"4d609a2212380eb63d308b56d42f467ddcf0dbf8d10dee1407d48b0fcdcd2862"}
Sep 09 20:56:17.113246 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.113117    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cg7cz\" (UniqueName: \"kubernetes.io/projected/ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9-kube-api-access-cg7cz\") pod \"ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9\" (UID: \"ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9\") "
Sep 09 20:56:17.113246 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.113167    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9-plugins-config-volume\") pod \"ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9\" (UID: \"ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9\") "
Sep 09 20:56:17.113790 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.113683    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pg2k\" (UniqueName: \"kubernetes.io/projected/01483de9-50af-4cbc-8d3e-373e8a6fc1f8-kube-api-access-7pg2k\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:56:17.113894 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.113864    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9" (UID: "ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:56:17.122192 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.122162    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9-kube-api-access-cg7cz" (OuterVolumeSpecName: "kube-api-access-cg7cz") pod "ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9" (UID: "ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9"). InnerVolumeSpecName "kube-api-access-cg7cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:56:17.125624 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.125595    2556 scope.go:117] "RemoveContainer" containerID="a34733430582e04c45975c5d5ad4912c8838a3f12ecdb4cd9289b6aa43ec2d75"
Sep 09 20:56:17.126312 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:56:17.126290    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a34733430582e04c45975c5d5ad4912c8838a3f12ecdb4cd9289b6aa43ec2d75\": container with ID starting with a34733430582e04c45975c5d5ad4912c8838a3f12ecdb4cd9289b6aa43ec2d75 not found: ID does not exist" containerID="a34733430582e04c45975c5d5ad4912c8838a3f12ecdb4cd9289b6aa43ec2d75"
Sep 09 20:56:17.126394 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.126320    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a34733430582e04c45975c5d5ad4912c8838a3f12ecdb4cd9289b6aa43ec2d75"} err="failed to get container status \"a34733430582e04c45975c5d5ad4912c8838a3f12ecdb4cd9289b6aa43ec2d75\": rpc error: code = NotFound desc = could not find container \"a34733430582e04c45975c5d5ad4912c8838a3f12ecdb4cd9289b6aa43ec2d75\": container with ID starting with a34733430582e04c45975c5d5ad4912c8838a3f12ecdb4cd9289b6aa43ec2d75 not found: ID does not exist"
Sep 09 20:56:17.126394 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.126338    2556 scope.go:117] "RemoveContainer" containerID="4fdf591f4c29c0a153f0fe318838d16a00afba7f8c39a4e646175d94109ef162"
Sep 09 20:56:17.130051 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.130028    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf"]
Sep 09 20:56:17.133473 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.133451    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fbe7be47-7d9ff5f7f7-cwqzf"]
Sep 09 20:56:17.135379 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.135361    2556 scope.go:117] "RemoveContainer" containerID="4fdf591f4c29c0a153f0fe318838d16a00afba7f8c39a4e646175d94109ef162"
Sep 09 20:56:17.135622 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:56:17.135602    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fdf591f4c29c0a153f0fe318838d16a00afba7f8c39a4e646175d94109ef162\": container with ID starting with 4fdf591f4c29c0a153f0fe318838d16a00afba7f8c39a4e646175d94109ef162 not found: ID does not exist" containerID="4fdf591f4c29c0a153f0fe318838d16a00afba7f8c39a4e646175d94109ef162"
Sep 09 20:56:17.135667 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.135628    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fdf591f4c29c0a153f0fe318838d16a00afba7f8c39a4e646175d94109ef162"} err="failed to get container status \"4fdf591f4c29c0a153f0fe318838d16a00afba7f8c39a4e646175d94109ef162\": rpc error: code = NotFound desc = could not find container \"4fdf591f4c29c0a153f0fe318838d16a00afba7f8c39a4e646175d94109ef162\": container with ID starting with 4fdf591f4c29c0a153f0fe318838d16a00afba7f8c39a4e646175d94109ef162 not found: ID does not exist"
Sep 09 20:56:17.214946 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.214925    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg7cz\" (UniqueName: \"kubernetes.io/projected/ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9-kube-api-access-cg7cz\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:56:17.214946 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.214945    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9-plugins-config-volume\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:56:17.430238 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.430205    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l"]
Sep 09 20:56:17.433860 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.433834    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fbe7be47-5cc978c9cd-kgd4l"]
Sep 09 20:56:17.805208 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.805176    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:17.805544 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.805218    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:17.817729 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:17.817706    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:18.126572 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:18.126487    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 20:56:18.893289 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:18.893255    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01483de9-50af-4cbc-8d3e-373e8a6fc1f8" path="/var/lib/kubelet/pods/01483de9-50af-4cbc-8d3e-373e8a6fc1f8/volumes"
Sep 09 20:56:18.893658 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:18.893577    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b2c37cb-0639-4a21-87a2-1abcf7214f29" path="/var/lib/kubelet/pods/6b2c37cb-0639-4a21-87a2-1abcf7214f29/volumes"
Sep 09 20:56:18.893897 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:18.893884    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9" path="/var/lib/kubelet/pods/ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9/volumes"
Sep 09 20:56:23.601316 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:23.601272    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:23.601836 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:23.601328    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:23.614179 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:23.614148    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:24.158098 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:24.158064    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:56:48.111806 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:56:48.111770    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:57:51.900266 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:51.900230    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:57:55.102871 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.102834    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"]
Sep 09 20:57:55.103320 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.103304    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01483de9-50af-4cbc-8d3e-373e8a6fc1f8" containerName="payload-processing"
Sep 09 20:57:55.103379 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.103321    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="01483de9-50af-4cbc-8d3e-373e8a6fc1f8" containerName="payload-processing"
Sep 09 20:57:55.103379 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.103361    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9" containerName="payload-pre-processing"
Sep 09 20:57:55.103379 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.103367    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9" containerName="payload-pre-processing"
Sep 09 20:57:55.103486 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.103430    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca9a7fc7-8c1e-4f75-b927-9b685f81a4a9" containerName="payload-pre-processing"
Sep 09 20:57:55.103486 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.103438    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="01483de9-50af-4cbc-8d3e-373e8a6fc1f8" containerName="payload-processing"
Sep 09 20:57:55.105563 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.105547    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:57:55.109714 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.109690    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-a2fb5a\"/\"default-dockercfg-7v86l\""
Sep 09 20:57:55.111726 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.111705    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-a2fb5a\"/\"kube-root-ca.crt\""
Sep 09 20:57:55.111950 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.111933    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-a2fb5a\"/\"openshift-service-ca.crt\""
Sep 09 20:57:55.112027 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.111994    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-a2fb5a\"/\"auth-test-model-a2fb5a-kserve-self-signed-certs\""
Sep 09 20:57:55.120435 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.120410    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"]
Sep 09 20:57:55.227919 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.227878    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-home\") pod \"auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj\" (UID: \"06060449-719b-4544-a976-30268aede890\") " pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:57:55.228029 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.227978    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-model-cache\") pod \"auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj\" (UID: \"06060449-719b-4544-a976-30268aede890\") " pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:57:55.228029 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.228013    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06060449-719b-4544-a976-30268aede890-tls-certs\") pod \"auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj\" (UID: \"06060449-719b-4544-a976-30268aede890\") " pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:57:55.228109 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.228032    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-dshm\") pod \"auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj\" (UID: \"06060449-719b-4544-a976-30268aede890\") " pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:57:55.228109 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.228057    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-tmp-dir\") pod \"auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj\" (UID: \"06060449-719b-4544-a976-30268aede890\") " pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:57:55.228190 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.228171    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mtfx\" (UniqueName: \"kubernetes.io/projected/06060449-719b-4544-a976-30268aede890-kube-api-access-5mtfx\") pod \"auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj\" (UID: \"06060449-719b-4544-a976-30268aede890\") " pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:57:55.328879 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.328846    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06060449-719b-4544-a976-30268aede890-tls-certs\") pod \"auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj\" (UID: \"06060449-719b-4544-a976-30268aede890\") " pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:57:55.328879 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.328884    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-dshm\") pod \"auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj\" (UID: \"06060449-719b-4544-a976-30268aede890\") " pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:57:55.329102 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.328932    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-tmp-dir\") pod \"auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj\" (UID: \"06060449-719b-4544-a976-30268aede890\") " pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:57:55.329102 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.328988    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mtfx\" (UniqueName: \"kubernetes.io/projected/06060449-719b-4544-a976-30268aede890-kube-api-access-5mtfx\") pod \"auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj\" (UID: \"06060449-719b-4544-a976-30268aede890\") " pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:57:55.329102 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.329042    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-home\") pod \"auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj\" (UID: \"06060449-719b-4544-a976-30268aede890\") " pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:57:55.329102 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.329086    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-model-cache\") pod \"auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj\" (UID: \"06060449-719b-4544-a976-30268aede890\") " pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:57:55.329439 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.329406    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-home\") pod \"auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj\" (UID: \"06060449-719b-4544-a976-30268aede890\") " pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:57:55.329439 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.329418    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-tmp-dir\") pod \"auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj\" (UID: \"06060449-719b-4544-a976-30268aede890\") " pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:57:55.329584 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.329456    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-model-cache\") pod \"auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj\" (UID: \"06060449-719b-4544-a976-30268aede890\") " pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:57:55.331169 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.331151    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-dshm\") pod \"auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj\" (UID: \"06060449-719b-4544-a976-30268aede890\") " pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:57:55.331574 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.331552    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06060449-719b-4544-a976-30268aede890-tls-certs\") pod \"auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj\" (UID: \"06060449-719b-4544-a976-30268aede890\") " pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:57:55.336785 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.336762    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mtfx\" (UniqueName: \"kubernetes.io/projected/06060449-719b-4544-a976-30268aede890-kube-api-access-5mtfx\") pod \"auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj\" (UID: \"06060449-719b-4544-a976-30268aede890\") " pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:57:55.416066 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.416002    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:57:55.686440 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:55.686412    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"]
Sep 09 20:57:55.688014 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:57:55.687983    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06060449_719b_4544_a976_30268aede890.slice/crio-977f584ae9b532466d0f781af1c969124b0728f0c1d9ccbb59d6421af4aac1d3 WatchSource:0}: Error finding container 977f584ae9b532466d0f781af1c969124b0728f0c1d9ccbb59d6421af4aac1d3: Status 404 returned error can't find the container with id 977f584ae9b532466d0f781af1c969124b0728f0c1d9ccbb59d6421af4aac1d3
Sep 09 20:57:56.660487 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:56.660451    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj" event={"ID":"06060449-719b-4544-a976-30268aede890","Type":"ContainerStarted","Data":"815a84a69fa8a3bd088c2b33d3e369dabbced367abd5f511b3659d8f2552a070"}
Sep 09 20:57:56.660885 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:56.660503    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj" event={"ID":"06060449-719b-4544-a976-30268aede890","Type":"ContainerStarted","Data":"977f584ae9b532466d0f781af1c969124b0728f0c1d9ccbb59d6421af4aac1d3"}
Sep 09 20:57:56.678137 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:57:56.678087    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj" podStartSLOduration=1.678070752 podStartE2EDuration="1.678070752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:57:56.676268521 +0000 UTC m=+1522.356228841" watchObservedRunningTime="2026-09-09 20:57:56.678070752 +0000 UTC m=+1522.358031062"
Sep 09 20:58:05.416706 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:05.416630    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:58:05.416706 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:05.416675    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:58:05.429131 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:05.429108    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:58:05.728843 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:05.728771    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 20:58:08.365291 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.365258    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"]
Sep 09 20:58:08.367786 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.367768    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.370774 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.370753    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ada93f-openshift-default-dockercfg-pmjcj\""
Sep 09 20:58:08.382937 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.382900    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"]
Sep 09 20:58:08.445728 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.445695    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5275b14-3ea3-45a3-b363-b203dc340646-istio-token\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.445928 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.445815    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-istio-envoy\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.445928 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.445843    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5275b14-3ea3-45a3-b363-b203dc340646-istiod-ca-cert\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.445928 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.445878    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-credential-socket\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.446127 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.445934    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-workload-certs\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.446127 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.446032    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-workload-socket\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.446127 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.446055    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-istio-data\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.446127 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.446090    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5275b14-3ea3-45a3-b363-b203dc340646-istio-podinfo\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.446375 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.446141    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs7qf\" (UniqueName: \"kubernetes.io/projected/c5275b14-3ea3-45a3-b363-b203dc340646-kube-api-access-xs7qf\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.547102 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.547059    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5275b14-3ea3-45a3-b363-b203dc340646-istio-token\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.547272 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.547151    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-istio-envoy\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.547272 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.547179    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5275b14-3ea3-45a3-b363-b203dc340646-istiod-ca-cert\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.547272 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.547210    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-credential-socket\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.547272 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.547234    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-workload-certs\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.547489 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.547308    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-workload-socket\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.547489 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.547329    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-istio-data\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.547489 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.547469    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5275b14-3ea3-45a3-b363-b203dc340646-istio-podinfo\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.547652 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.547512    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs7qf\" (UniqueName: \"kubernetes.io/projected/c5275b14-3ea3-45a3-b363-b203dc340646-kube-api-access-xs7qf\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.547714 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.547685    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-credential-socket\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.547771 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.547752    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-workload-certs\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.547829 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.547806    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-workload-socket\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.547948 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.547901    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-istio-data\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.548031 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.548013    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5275b14-3ea3-45a3-b363-b203dc340646-istiod-ca-cert\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.549534 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.549512    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-istio-envoy\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.549990 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.549973    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5275b14-3ea3-45a3-b363-b203dc340646-istio-podinfo\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.555141 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.555118    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5275b14-3ea3-45a3-b363-b203dc340646-istio-token\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.555660 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.555641    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs7qf\" (UniqueName: \"kubernetes.io/projected/c5275b14-3ea3-45a3-b363-b203dc340646-kube-api-access-xs7qf\") pod \"e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") " pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.678401 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.678338    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:08.955741 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.955716    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"]
Sep 09 20:58:08.958062 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:58:08.958035    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5275b14_3ea3_45a3_b363_b203dc340646.slice/crio-77db2b9a0012808dab162cc3836b6c74d03e050924bad8c8b9ac16930f60000a WatchSource:0}: Error finding container 77db2b9a0012808dab162cc3836b6c74d03e050924bad8c8b9ac16930f60000a: Status 404 returned error can't find the container with id 77db2b9a0012808dab162cc3836b6c74d03e050924bad8c8b9ac16930f60000a
Sep 09 20:58:08.960319 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.960285    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:58:08.960400 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.960350    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:58:08.960400 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:08.960385    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:58:09.743648 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:09.743614    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s" event={"ID":"c5275b14-3ea3-45a3-b363-b203dc340646","Type":"ContainerStarted","Data":"1031d6af467c0e1e7bc728439f3aba764a56eed71cb9e2b85c503e014e800f88"}
Sep 09 20:58:09.744038 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:09.743666    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s" event={"ID":"c5275b14-3ea3-45a3-b363-b203dc340646","Type":"ContainerStarted","Data":"77db2b9a0012808dab162cc3836b6c74d03e050924bad8c8b9ac16930f60000a"}
Sep 09 20:58:09.769290 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:09.769246    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s" podStartSLOduration=1.769233563 podStartE2EDuration="1.769233563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:09.766300128 +0000 UTC m=+1535.446260441" watchObservedRunningTime="2026-09-09 20:58:09.769233563 +0000 UTC m=+1535.449193871"
Sep 09 20:58:10.679125 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:10.679089    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:10.684454 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:10.684424    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:10.749806 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:10.749780    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:10.750586 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:10.750568    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:13.758014 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:13.757981    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n"]
Sep 09 20:58:13.761041 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:13.761013    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n"
Sep 09 20:58:13.763890 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:13.763865    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ada93f-dockercfg-ktt98\""
Sep 09 20:58:13.764070 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:13.763871    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ada93f\""
Sep 09 20:58:13.773035 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:13.772977    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n"]
Sep 09 20:58:13.898825 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:13.898783    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5qvn\" (UniqueName: \"kubernetes.io/projected/0e9b3209-3035-4e20-be75-dfa8731efb0f-kube-api-access-b5qvn\") pod \"payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n\" (UID: \"0e9b3209-3035-4e20-be75-dfa8731efb0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n"
Sep 09 20:58:13.899039 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:13.898856    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9b3209-3035-4e20-be75-dfa8731efb0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n\" (UID: \"0e9b3209-3035-4e20-be75-dfa8731efb0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n"
Sep 09 20:58:14.000382 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:14.000345    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5qvn\" (UniqueName: \"kubernetes.io/projected/0e9b3209-3035-4e20-be75-dfa8731efb0f-kube-api-access-b5qvn\") pod \"payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n\" (UID: \"0e9b3209-3035-4e20-be75-dfa8731efb0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n"
Sep 09 20:58:14.000541 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:14.000396    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9b3209-3035-4e20-be75-dfa8731efb0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n\" (UID: \"0e9b3209-3035-4e20-be75-dfa8731efb0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n"
Sep 09 20:58:14.001098 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:14.001080    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9b3209-3035-4e20-be75-dfa8731efb0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n\" (UID: \"0e9b3209-3035-4e20-be75-dfa8731efb0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n"
Sep 09 20:58:14.009680 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:14.009626    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5qvn\" (UniqueName: \"kubernetes.io/projected/0e9b3209-3035-4e20-be75-dfa8731efb0f-kube-api-access-b5qvn\") pod \"payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n\" (UID: \"0e9b3209-3035-4e20-be75-dfa8731efb0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n"
Sep 09 20:58:14.073622 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:14.073591    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n"
Sep 09 20:58:14.344072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:14.344049    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n"]
Sep 09 20:58:14.346149 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:58:14.346125    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9b3209_3035_4e20_be75_dfa8731efb0f.slice/crio-3d636b0fca938e29bf2e4b91d34ef0d1745c87db3a8c052b7924d6dddee28cd8 WatchSource:0}: Error finding container 3d636b0fca938e29bf2e4b91d34ef0d1745c87db3a8c052b7924d6dddee28cd8: Status 404 returned error can't find the container with id 3d636b0fca938e29bf2e4b91d34ef0d1745c87db3a8c052b7924d6dddee28cd8
Sep 09 20:58:14.773000 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:14.772962    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n" event={"ID":"0e9b3209-3035-4e20-be75-dfa8731efb0f","Type":"ContainerStarted","Data":"d7cbbaa53ba14b308cfe1220d10a7a7158cade4e6622e337adfc3df6bd90f174"}
Sep 09 20:58:14.773430 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:14.773024    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n" event={"ID":"0e9b3209-3035-4e20-be75-dfa8731efb0f","Type":"ContainerStarted","Data":"3d636b0fca938e29bf2e4b91d34ef0d1745c87db3a8c052b7924d6dddee28cd8"}
Sep 09 20:58:14.773430 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:14.773049    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n"
Sep 09 20:58:14.794480 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:14.794438    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n" podStartSLOduration=1.7944226890000001 podStartE2EDuration="1.794422689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:14.793528849 +0000 UTC m=+1540.473489173" watchObservedRunningTime="2026-09-09 20:58:14.794422689 +0000 UTC m=+1540.474382997"
Sep 09 20:58:25.781720 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:25.781690    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n"
Sep 09 20:58:26.308282 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:26.308247    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:58:36.901811 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:36.901730    2556 generic.go:356] "Generic (PLEG): container finished" podID="c5275b14-3ea3-45a3-b363-b203dc340646" containerID="1031d6af467c0e1e7bc728439f3aba764a56eed71cb9e2b85c503e014e800f88" exitCode=0
Sep 09 20:58:36.902213 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:36.901802    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s" event={"ID":"c5275b14-3ea3-45a3-b363-b203dc340646","Type":"ContainerDied","Data":"1031d6af467c0e1e7bc728439f3aba764a56eed71cb9e2b85c503e014e800f88"}
Sep 09 20:58:36.902213 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:36.902194    2556 scope.go:117] "RemoveContainer" containerID="1031d6af467c0e1e7bc728439f3aba764a56eed71cb9e2b85c503e014e800f88"
Sep 09 20:58:36.903437 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:36.903412    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:58:36.903495 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:36.903477    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:58:36.903532 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:36.903507    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:58:37.909060 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:37.909025    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s" event={"ID":"c5275b14-3ea3-45a3-b363-b203dc340646","Type":"ContainerStarted","Data":"655c733148d2ec755cf22ea3d1f706f96fb75882dec42e32a8471f18aefe6aea"}
Sep 09 20:58:38.678875 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:38.678780    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:38.680473 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:38.680433    2556 patch_prober.go:28] interesting pod/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 09 20:58:38.680696 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:38.680671    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s" podUID="c5275b14-3ea3-45a3-b363-b203dc340646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 09 20:58:39.679702 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:39.679667    2556 patch_prober.go:28] interesting pod/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 09 20:58:39.680302 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:39.680269    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s" podUID="c5275b14-3ea3-45a3-b363-b203dc340646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 09 20:58:40.678723 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:40.678686    2556 patch_prober.go:28] interesting pod/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 09 20:58:40.679026 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:40.678996    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s" podUID="c5275b14-3ea3-45a3-b363-b203dc340646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 09 20:58:40.749719 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:40.749680    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:41.679441 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:41.678969    2556 patch_prober.go:28] interesting pod/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 09 20:58:41.679441 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:41.679036    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s" podUID="c5275b14-3ea3-45a3-b363-b203dc340646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 09 20:58:42.494709 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:42.494675    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw"]
Sep 09 20:58:42.498006 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:42.497974    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw"
Sep 09 20:58:42.502944 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:42.502920    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4e2f93\""
Sep 09 20:58:42.511171 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:42.511149    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw"]
Sep 09 20:58:42.593575 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:42.593537    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v7m5\" (UniqueName: \"kubernetes.io/projected/3d4a2d20-44d6-438b-8471-722d4d0e3b81-kube-api-access-4v7m5\") pod \"payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw\" (UID: \"3d4a2d20-44d6-438b-8471-722d4d0e3b81\") " pod="openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw"
Sep 09 20:58:42.595469 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:42.595444    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d4a2d20-44d6-438b-8471-722d4d0e3b81-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw\" (UID: \"3d4a2d20-44d6-438b-8471-722d4d0e3b81\") " pod="openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw"
Sep 09 20:58:42.685928 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:42.680256    2556 patch_prober.go:28] interesting pod/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 09 20:58:42.685928 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:42.680315    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s" podUID="c5275b14-3ea3-45a3-b363-b203dc340646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 09 20:58:42.696680 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:42.696652    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v7m5\" (UniqueName: \"kubernetes.io/projected/3d4a2d20-44d6-438b-8471-722d4d0e3b81-kube-api-access-4v7m5\") pod \"payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw\" (UID: \"3d4a2d20-44d6-438b-8471-722d4d0e3b81\") " pod="openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw"
Sep 09 20:58:42.696814 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:42.696745    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d4a2d20-44d6-438b-8471-722d4d0e3b81-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw\" (UID: \"3d4a2d20-44d6-438b-8471-722d4d0e3b81\") " pod="openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw"
Sep 09 20:58:42.697424 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:42.697405    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d4a2d20-44d6-438b-8471-722d4d0e3b81-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw\" (UID: \"3d4a2d20-44d6-438b-8471-722d4d0e3b81\") " pod="openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw"
Sep 09 20:58:42.706351 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:42.706321    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v7m5\" (UniqueName: \"kubernetes.io/projected/3d4a2d20-44d6-438b-8471-722d4d0e3b81-kube-api-access-4v7m5\") pod \"payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw\" (UID: \"3d4a2d20-44d6-438b-8471-722d4d0e3b81\") " pod="openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw"
Sep 09 20:58:42.808848 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:42.808760    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw"
Sep 09 20:58:43.082372 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:43.082280    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw"]
Sep 09 20:58:43.682960 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:43.682899    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:43.684081 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:43.684061    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:58:43.951920 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:43.951814    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw" event={"ID":"3d4a2d20-44d6-438b-8471-722d4d0e3b81","Type":"ContainerStarted","Data":"84ba98455d34bb7ab1d0333cc9c5a0370c8fdb249897c73f5cd59a097f32d2ef"}
Sep 09 20:58:43.951920 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:43.951863    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw" event={"ID":"3d4a2d20-44d6-438b-8471-722d4d0e3b81","Type":"ContainerStarted","Data":"12e22397aae15bcd4b010ce8cbf02a1f1d96ec7e738acaca77c6c8c484be5959"}
Sep 09 20:58:43.974286 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:43.974235    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw" podStartSLOduration=1.974215789 podStartE2EDuration="1.974215789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:43.972010011 +0000 UTC m=+1569.651970318" watchObservedRunningTime="2026-09-09 20:58:43.974215789 +0000 UTC m=+1569.654176101"
Sep 09 20:58:44.957397 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:44.957364    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw"
Sep 09 20:58:55.964177 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:58:55.964150    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw"
Sep 09 20:59:08.252264 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:08.252222    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada93f-pq7qn"]
Sep 09 20:59:08.257319 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:08.257291    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada93f-pq7qn"
Sep 09 20:59:08.260869 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:08.260843    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsks9\""
Sep 09 20:59:08.261821 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:08.261797    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada93f-pq7qn"]
Sep 09 20:59:08.324644 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:08.324615    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20af2135-1f12-4d16-ba38-0f740c658ebd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ada93f-pq7qn\" (UID: \"20af2135-1f12-4d16-ba38-0f740c658ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada93f-pq7qn"
Sep 09 20:59:08.425434 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:08.425404    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20af2135-1f12-4d16-ba38-0f740c658ebd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ada93f-pq7qn\" (UID: \"20af2135-1f12-4d16-ba38-0f740c658ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada93f-pq7qn"
Sep 09 20:59:08.426054 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:08.426033    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20af2135-1f12-4d16-ba38-0f740c658ebd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ada93f-pq7qn\" (UID: \"20af2135-1f12-4d16-ba38-0f740c658ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada93f-pq7qn"
Sep 09 20:59:08.570238 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:08.570169    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada93f-pq7qn"
Sep 09 20:59:08.842402 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:08.842368    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada93f-pq7qn"]
Sep 09 20:59:08.844083 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:59:08.844056    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20af2135_1f12_4d16_ba38_0f740c658ebd.slice/crio-537dacbe321902ab9bc4d74a88adb3c75a506833b516250b0cb91889790dd800 WatchSource:0}: Error finding container 537dacbe321902ab9bc4d74a88adb3c75a506833b516250b0cb91889790dd800: Status 404 returned error can't find the container with id 537dacbe321902ab9bc4d74a88adb3c75a506833b516250b0cb91889790dd800
Sep 09 20:59:09.093381 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:09.093309    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada93f-pq7qn" event={"ID":"20af2135-1f12-4d16-ba38-0f740c658ebd","Type":"ContainerStarted","Data":"537dacbe321902ab9bc4d74a88adb3c75a506833b516250b0cb91889790dd800"}
Sep 09 20:59:10.100566 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:10.100537    2556 generic.go:356] "Generic (PLEG): container finished" podID="20af2135-1f12-4d16-ba38-0f740c658ebd" containerID="b79c8b4a51e68ef350edf7d5f7ea1f7827d83dd43572519f520551f000b9e3d2" exitCode=0
Sep 09 20:59:10.100969 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:10.100631    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada93f-pq7qn" event={"ID":"20af2135-1f12-4d16-ba38-0f740c658ebd","Type":"ContainerDied","Data":"b79c8b4a51e68ef350edf7d5f7ea1f7827d83dd43572519f520551f000b9e3d2"}
Sep 09 20:59:11.243032 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.243006    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada93f-pq7qn"
Sep 09 20:59:11.289731 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.289701    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"]
Sep 09 20:59:11.290224 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.290209    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20af2135-1f12-4d16-ba38-0f740c658ebd" containerName="revoke-keys"
Sep 09 20:59:11.290279 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.290228    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="20af2135-1f12-4d16-ba38-0f740c658ebd" containerName="revoke-keys"
Sep 09 20:59:11.290327 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.290316    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="20af2135-1f12-4d16-ba38-0f740c658ebd" containerName="revoke-keys"
Sep 09 20:59:11.292620 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.292601    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:11.297277 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.297254    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-933e9d\"/\"e2e-conflict-a-dff9f838-kserve-self-signed-certs\""
Sep 09 20:59:11.303970 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.303952    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"]
Sep 09 20:59:11.355511 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.355481    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20af2135-1f12-4d16-ba38-0f740c658ebd-maas-api-service-ca\") pod \"20af2135-1f12-4d16-ba38-0f740c658ebd\" (UID: \"20af2135-1f12-4d16-ba38-0f740c658ebd\") "
Sep 09 20:59:11.355700 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.355685    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-dshm\") pod \"e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") " pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:11.355815 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.355796    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20af2135-1f12-4d16-ba38-0f740c658ebd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "20af2135-1f12-4d16-ba38-0f740c658ebd" (UID: "20af2135-1f12-4d16-ba38-0f740c658ebd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:59:11.355859 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.355809    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ee8777-c711-4e59-ab09-91eb3d62d56d-tls-certs\") pod \"e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") " pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:11.355859 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.355853    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-tmp-dir\") pod \"e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") " pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:11.355959 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.355875    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-model-cache\") pod \"e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") " pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:11.355959 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.355891    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5pb6\" (UniqueName: \"kubernetes.io/projected/40ee8777-c711-4e59-ab09-91eb3d62d56d-kube-api-access-h5pb6\") pod \"e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") " pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:11.356035 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.355966    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-home\") pod \"e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") " pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:11.356072 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.356036    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20af2135-1f12-4d16-ba38-0f740c658ebd-maas-api-service-ca\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:11.456841 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.456777    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ee8777-c711-4e59-ab09-91eb3d62d56d-tls-certs\") pod \"e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") " pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:11.456841 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.456815    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-tmp-dir\") pod \"e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") " pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:11.456841 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.456832    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-model-cache\") pod \"e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") " pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:11.457098 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.456851    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5pb6\" (UniqueName: \"kubernetes.io/projected/40ee8777-c711-4e59-ab09-91eb3d62d56d-kube-api-access-h5pb6\") pod \"e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") " pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:11.457098 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.457042    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-home\") pod \"e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") " pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:11.457209 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.457159    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-dshm\") pod \"e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") " pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:11.457262 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.457203    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-tmp-dir\") pod \"e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") " pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:11.457262 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.457232    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-model-cache\") pod \"e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") " pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:11.457373 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.457322    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-home\") pod \"e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") " pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:11.459138 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.459117    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-dshm\") pod \"e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") " pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:11.459242 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.459227    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ee8777-c711-4e59-ab09-91eb3d62d56d-tls-certs\") pod \"e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") " pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:11.465899 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.465873    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5pb6\" (UniqueName: \"kubernetes.io/projected/40ee8777-c711-4e59-ab09-91eb3d62d56d-kube-api-access-h5pb6\") pod \"e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") " pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:11.602397 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.602363    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:11.876930 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:11.876879    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"]
Sep 09 20:59:11.877938 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:59:11.877896    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40ee8777_c711_4e59_ab09_91eb3d62d56d.slice/crio-b7d28df7b581f361be1d7d5dff4008f2457811095ce436a13de9262655565bcb WatchSource:0}: Error finding container b7d28df7b581f361be1d7d5dff4008f2457811095ce436a13de9262655565bcb: Status 404 returned error can't find the container with id b7d28df7b581f361be1d7d5dff4008f2457811095ce436a13de9262655565bcb
Sep 09 20:59:12.113154 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:12.113120    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj" event={"ID":"40ee8777-c711-4e59-ab09-91eb3d62d56d","Type":"ContainerStarted","Data":"2f02d9a5fe275bb2f73c7906e02a049a7b736bdde02c5a152267306380b86353"}
Sep 09 20:59:12.113316 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:12.113161    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj" event={"ID":"40ee8777-c711-4e59-ab09-91eb3d62d56d","Type":"ContainerStarted","Data":"b7d28df7b581f361be1d7d5dff4008f2457811095ce436a13de9262655565bcb"}
Sep 09 20:59:12.114486 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:12.114466    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada93f-pq7qn"
Sep 09 20:59:12.114593 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:12.114485    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada93f-pq7qn" event={"ID":"20af2135-1f12-4d16-ba38-0f740c658ebd","Type":"ContainerDied","Data":"537dacbe321902ab9bc4d74a88adb3c75a506833b516250b0cb91889790dd800"}
Sep 09 20:59:12.114593 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:12.114521    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="537dacbe321902ab9bc4d74a88adb3c75a506833b516250b0cb91889790dd800"
Sep 09 20:59:12.134499 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:12.134426    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj" podStartSLOduration=1.134413773 podStartE2EDuration="1.134413773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:59:12.132526949 +0000 UTC m=+1597.812487258" watchObservedRunningTime="2026-09-09 20:59:12.134413773 +0000 UTC m=+1597.814374083"
Sep 09 20:59:21.603341 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:21.603295    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:21.603341 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:21.603347    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:21.615870 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:21.615842    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:22.181992 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:22.181960    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 20:59:23.971013 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:23.970968    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n"]
Sep 09 20:59:23.971470 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:23.971217    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n" podUID="0e9b3209-3035-4e20-be75-dfa8731efb0f" containerName="payload-pre-processing" containerID="cri-o://d7cbbaa53ba14b308cfe1220d10a7a7158cade4e6622e337adfc3df6bd90f174" gracePeriod=30
Sep 09 20:59:24.155394 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:24.155324    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada93f-pq7qn"]
Sep 09 20:59:24.158792 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:24.158765    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ada93f-pq7qn"]
Sep 09 20:59:24.183898 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:24.183870    2556 generic.go:356] "Generic (PLEG): container finished" podID="0e9b3209-3035-4e20-be75-dfa8731efb0f" containerID="d7cbbaa53ba14b308cfe1220d10a7a7158cade4e6622e337adfc3df6bd90f174" exitCode=0
Sep 09 20:59:24.184041 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:24.183941    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n" event={"ID":"0e9b3209-3035-4e20-be75-dfa8731efb0f","Type":"ContainerDied","Data":"d7cbbaa53ba14b308cfe1220d10a7a7158cade4e6622e337adfc3df6bd90f174"}
Sep 09 20:59:24.226959 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:24.226880    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n"
Sep 09 20:59:24.376899 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:24.376863    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5qvn\" (UniqueName: \"kubernetes.io/projected/0e9b3209-3035-4e20-be75-dfa8731efb0f-kube-api-access-b5qvn\") pod \"0e9b3209-3035-4e20-be75-dfa8731efb0f\" (UID: \"0e9b3209-3035-4e20-be75-dfa8731efb0f\") "
Sep 09 20:59:24.377119 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:24.377050    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9b3209-3035-4e20-be75-dfa8731efb0f-plugins-config-volume\") pod \"0e9b3209-3035-4e20-be75-dfa8731efb0f\" (UID: \"0e9b3209-3035-4e20-be75-dfa8731efb0f\") "
Sep 09 20:59:24.377469 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:24.377441    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e9b3209-3035-4e20-be75-dfa8731efb0f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0e9b3209-3035-4e20-be75-dfa8731efb0f" (UID: "0e9b3209-3035-4e20-be75-dfa8731efb0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:59:24.379172 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:24.379145    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9b3209-3035-4e20-be75-dfa8731efb0f-kube-api-access-b5qvn" (OuterVolumeSpecName: "kube-api-access-b5qvn") pod "0e9b3209-3035-4e20-be75-dfa8731efb0f" (UID: "0e9b3209-3035-4e20-be75-dfa8731efb0f"). InnerVolumeSpecName "kube-api-access-b5qvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:59:24.478379 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:24.478299    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9b3209-3035-4e20-be75-dfa8731efb0f-plugins-config-volume\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:24.478379 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:24.478339    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5qvn\" (UniqueName: \"kubernetes.io/projected/0e9b3209-3035-4e20-be75-dfa8731efb0f-kube-api-access-b5qvn\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:24.893684 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:24.893654    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20af2135-1f12-4d16-ba38-0f740c658ebd" path="/var/lib/kubelet/pods/20af2135-1f12-4d16-ba38-0f740c658ebd/volumes"
Sep 09 20:59:25.190538 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:25.190467    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n"
Sep 09 20:59:25.190538 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:25.190497    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n" event={"ID":"0e9b3209-3035-4e20-be75-dfa8731efb0f","Type":"ContainerDied","Data":"3d636b0fca938e29bf2e4b91d34ef0d1745c87db3a8c052b7924d6dddee28cd8"}
Sep 09 20:59:25.190996 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:25.190553    2556 scope.go:117] "RemoveContainer" containerID="d7cbbaa53ba14b308cfe1220d10a7a7158cade4e6622e337adfc3df6bd90f174"
Sep 09 20:59:25.207157 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:25.207133    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n"]
Sep 09 20:59:25.211151 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:25.211127    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ada93f-6644c6f586-qvp5n"]
Sep 09 20:59:26.893592 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:26.893558    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e9b3209-3035-4e20-be75-dfa8731efb0f" path="/var/lib/kubelet/pods/0e9b3209-3035-4e20-be75-dfa8731efb0f/volumes"
Sep 09 20:59:27.705158 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:27.705077    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 20:59:30.595755 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:30.595723    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"]
Sep 09 20:59:30.596238 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:30.595963    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s" podUID="c5275b14-3ea3-45a3-b363-b203dc340646" containerName="istio-proxy" containerID="cri-o://655c733148d2ec755cf22ea3d1f706f96fb75882dec42e32a8471f18aefe6aea" gracePeriod=30
Sep 09 20:59:30.937635 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:30.937544    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e2f93-5nvfx"]
Sep 09 20:59:30.938124 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:30.938103    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e9b3209-3035-4e20-be75-dfa8731efb0f" containerName="payload-pre-processing"
Sep 09 20:59:30.938124 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:30.938125    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e9b3209-3035-4e20-be75-dfa8731efb0f" containerName="payload-pre-processing"
Sep 09 20:59:30.938299 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:30.938288    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e9b3209-3035-4e20-be75-dfa8731efb0f" containerName="payload-pre-processing"
Sep 09 20:59:30.941487 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:30.941461    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e2f93-5nvfx"
Sep 09 20:59:30.944211 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:30.944193    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsks9\""
Sep 09 20:59:30.950704 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:30.950680    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e2f93-5nvfx"]
Sep 09 20:59:31.042466 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:31.042438    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8a51ec-9ef7-4797-9b50-2b0abed94830-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4e2f93-5nvfx\" (UID: \"5b8a51ec-9ef7-4797-9b50-2b0abed94830\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e2f93-5nvfx"
Sep 09 20:59:31.143585 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:31.143562    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8a51ec-9ef7-4797-9b50-2b0abed94830-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4e2f93-5nvfx\" (UID: \"5b8a51ec-9ef7-4797-9b50-2b0abed94830\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e2f93-5nvfx"
Sep 09 20:59:31.144213 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:31.144194    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8a51ec-9ef7-4797-9b50-2b0abed94830-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4e2f93-5nvfx\" (UID: \"5b8a51ec-9ef7-4797-9b50-2b0abed94830\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e2f93-5nvfx"
Sep 09 20:59:31.254091 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:31.254067    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e2f93-5nvfx"
Sep 09 20:59:31.528536 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:31.528387    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e2f93-5nvfx"]
Sep 09 20:59:31.530936 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:59:31.530889    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8a51ec_9ef7_4797_9b50_2b0abed94830.slice/crio-ac822c0f0148dbf4a23a5c30b4f4d4ec2536f95e5013939bf227470c2d407e90 WatchSource:0}: Error finding container ac822c0f0148dbf4a23a5c30b4f4d4ec2536f95e5013939bf227470c2d407e90: Status 404 returned error can't find the container with id ac822c0f0148dbf4a23a5c30b4f4d4ec2536f95e5013939bf227470c2d407e90
Sep 09 20:59:32.235382 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:32.235348    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e2f93-5nvfx" event={"ID":"5b8a51ec-9ef7-4797-9b50-2b0abed94830","Type":"ContainerStarted","Data":"fcc0578ccedd5c38851b5646e825e64d85495928395b7cf674b719272f4c51bb"}
Sep 09 20:59:32.235862 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:32.235406    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e2f93-5nvfx" event={"ID":"5b8a51ec-9ef7-4797-9b50-2b0abed94830","Type":"ContainerStarted","Data":"ac822c0f0148dbf4a23a5c30b4f4d4ec2536f95e5013939bf227470c2d407e90"}
Sep 09 20:59:32.253218 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:32.253176    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e2f93-5nvfx" podStartSLOduration=2.2531630480000002 podStartE2EDuration="2.253163048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:59:32.251823593 +0000 UTC m=+1617.931783903" watchObservedRunningTime="2026-09-09 20:59:32.253163048 +0000 UTC m=+1617.933123356"
Sep 09 20:59:33.243101 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:33.243069    2556 generic.go:356] "Generic (PLEG): container finished" podID="5b8a51ec-9ef7-4797-9b50-2b0abed94830" containerID="fcc0578ccedd5c38851b5646e825e64d85495928395b7cf674b719272f4c51bb" exitCode=0
Sep 09 20:59:33.243492 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:33.243153    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e2f93-5nvfx" event={"ID":"5b8a51ec-9ef7-4797-9b50-2b0abed94830","Type":"ContainerDied","Data":"fcc0578ccedd5c38851b5646e825e64d85495928395b7cf674b719272f4c51bb"}
Sep 09 20:59:34.388837 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:34.388814    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e2f93-5nvfx"
Sep 09 20:59:34.576677 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:34.576598    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8a51ec-9ef7-4797-9b50-2b0abed94830-maas-api-service-ca\") pod \"5b8a51ec-9ef7-4797-9b50-2b0abed94830\" (UID: \"5b8a51ec-9ef7-4797-9b50-2b0abed94830\") "
Sep 09 20:59:34.577088 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:34.577068    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b8a51ec-9ef7-4797-9b50-2b0abed94830-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b8a51ec-9ef7-4797-9b50-2b0abed94830" (UID: "5b8a51ec-9ef7-4797-9b50-2b0abed94830"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:59:34.678455 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:34.678432    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8a51ec-9ef7-4797-9b50-2b0abed94830-maas-api-service-ca\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:35.258124 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.258097    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e2f93-5nvfx"
Sep 09 20:59:35.258273 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.258135    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e2f93-5nvfx" event={"ID":"5b8a51ec-9ef7-4797-9b50-2b0abed94830","Type":"ContainerDied","Data":"ac822c0f0148dbf4a23a5c30b4f4d4ec2536f95e5013939bf227470c2d407e90"}
Sep 09 20:59:35.258273 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.258182    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac822c0f0148dbf4a23a5c30b4f4d4ec2536f95e5013939bf227470c2d407e90"
Sep 09 20:59:35.860156 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.860104    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:59:35.886503 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.886477    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-istio-data\") pod \"c5275b14-3ea3-45a3-b363-b203dc340646\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") "
Sep 09 20:59:35.886650 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.886524    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5275b14-3ea3-45a3-b363-b203dc340646-istiod-ca-cert\") pod \"c5275b14-3ea3-45a3-b363-b203dc340646\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") "
Sep 09 20:59:35.886650 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.886585    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-workload-socket\") pod \"c5275b14-3ea3-45a3-b363-b203dc340646\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") "
Sep 09 20:59:35.886650 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.886633    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xs7qf\" (UniqueName: \"kubernetes.io/projected/c5275b14-3ea3-45a3-b363-b203dc340646-kube-api-access-xs7qf\") pod \"c5275b14-3ea3-45a3-b363-b203dc340646\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") "
Sep 09 20:59:35.886820 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.886685    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-istio-envoy\") pod \"c5275b14-3ea3-45a3-b363-b203dc340646\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") "
Sep 09 20:59:35.886820 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.886744    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-credential-socket\") pod \"c5275b14-3ea3-45a3-b363-b203dc340646\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") "
Sep 09 20:59:35.886820 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.886741    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-istio-data" (OuterVolumeSpecName: "istio-data") pod "c5275b14-3ea3-45a3-b363-b203dc340646" (UID: "c5275b14-3ea3-45a3-b363-b203dc340646"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:59:35.886820 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.886780    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-workload-certs\") pod \"c5275b14-3ea3-45a3-b363-b203dc340646\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") "
Sep 09 20:59:35.886820 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.886814    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5275b14-3ea3-45a3-b363-b203dc340646-istio-token\") pod \"c5275b14-3ea3-45a3-b363-b203dc340646\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") "
Sep 09 20:59:35.887119 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.886872    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5275b14-3ea3-45a3-b363-b203dc340646-istio-podinfo\") pod \"c5275b14-3ea3-45a3-b363-b203dc340646\" (UID: \"c5275b14-3ea3-45a3-b363-b203dc340646\") "
Sep 09 20:59:35.887119 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.886895    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c5275b14-3ea3-45a3-b363-b203dc340646" (UID: "c5275b14-3ea3-45a3-b363-b203dc340646"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:59:35.887119 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.886920    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5275b14-3ea3-45a3-b363-b203dc340646-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c5275b14-3ea3-45a3-b363-b203dc340646" (UID: "c5275b14-3ea3-45a3-b363-b203dc340646"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:59:35.887278 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.887128    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c5275b14-3ea3-45a3-b363-b203dc340646" (UID: "c5275b14-3ea3-45a3-b363-b203dc340646"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:59:35.887333 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.887311    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-workload-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:35.887398 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.887331    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-istio-data\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:35.887398 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.887345    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5275b14-3ea3-45a3-b363-b203dc340646-istiod-ca-cert\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:35.887398 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.887356    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-workload-socket\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:35.887564 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.887540    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c5275b14-3ea3-45a3-b363-b203dc340646" (UID: "c5275b14-3ea3-45a3-b363-b203dc340646"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:59:35.889348 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.889321    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c5275b14-3ea3-45a3-b363-b203dc340646" (UID: "c5275b14-3ea3-45a3-b363-b203dc340646"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:59:35.889657 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.889637    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5275b14-3ea3-45a3-b363-b203dc340646-istio-token" (OuterVolumeSpecName: "istio-token") pod "c5275b14-3ea3-45a3-b363-b203dc340646" (UID: "c5275b14-3ea3-45a3-b363-b203dc340646"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:59:35.889832 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.889674    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5275b14-3ea3-45a3-b363-b203dc340646-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c5275b14-3ea3-45a3-b363-b203dc340646" (UID: "c5275b14-3ea3-45a3-b363-b203dc340646"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 20:59:35.890763 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.890739    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5275b14-3ea3-45a3-b363-b203dc340646-kube-api-access-xs7qf" (OuterVolumeSpecName: "kube-api-access-xs7qf") pod "c5275b14-3ea3-45a3-b363-b203dc340646" (UID: "c5275b14-3ea3-45a3-b363-b203dc340646"). InnerVolumeSpecName "kube-api-access-xs7qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:59:35.988619 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.988595    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs7qf\" (UniqueName: \"kubernetes.io/projected/c5275b14-3ea3-45a3-b363-b203dc340646-kube-api-access-xs7qf\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:35.988619 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.988617    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-istio-envoy\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:35.988756 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.988626    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5275b14-3ea3-45a3-b363-b203dc340646-credential-socket\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:35.988756 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.988634    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5275b14-3ea3-45a3-b363-b203dc340646-istio-token\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:35.988756 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:35.988643    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5275b14-3ea3-45a3-b363-b203dc340646-istio-podinfo\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:36.265285 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:36.265251    2556 generic.go:356] "Generic (PLEG): container finished" podID="c5275b14-3ea3-45a3-b363-b203dc340646" containerID="655c733148d2ec755cf22ea3d1f706f96fb75882dec42e32a8471f18aefe6aea" exitCode=0
Sep 09 20:59:36.265444 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:36.265300    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"
Sep 09 20:59:36.265444 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:36.265332    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s" event={"ID":"c5275b14-3ea3-45a3-b363-b203dc340646","Type":"ContainerDied","Data":"655c733148d2ec755cf22ea3d1f706f96fb75882dec42e32a8471f18aefe6aea"}
Sep 09 20:59:36.265444 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:36.265386    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s" event={"ID":"c5275b14-3ea3-45a3-b363-b203dc340646","Type":"ContainerDied","Data":"77db2b9a0012808dab162cc3836b6c74d03e050924bad8c8b9ac16930f60000a"}
Sep 09 20:59:36.265444 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:36.265408    2556 scope.go:117] "RemoveContainer" containerID="655c733148d2ec755cf22ea3d1f706f96fb75882dec42e32a8471f18aefe6aea"
Sep 09 20:59:36.275958 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:36.275941    2556 scope.go:117] "RemoveContainer" containerID="1031d6af467c0e1e7bc728439f3aba764a56eed71cb9e2b85c503e014e800f88"
Sep 09 20:59:36.284196 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:36.284180    2556 scope.go:117] "RemoveContainer" containerID="655c733148d2ec755cf22ea3d1f706f96fb75882dec42e32a8471f18aefe6aea"
Sep 09 20:59:36.284471 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:59:36.284452    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"655c733148d2ec755cf22ea3d1f706f96fb75882dec42e32a8471f18aefe6aea\": container with ID starting with 655c733148d2ec755cf22ea3d1f706f96fb75882dec42e32a8471f18aefe6aea not found: ID does not exist" containerID="655c733148d2ec755cf22ea3d1f706f96fb75882dec42e32a8471f18aefe6aea"
Sep 09 20:59:36.284516 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:36.284479    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"655c733148d2ec755cf22ea3d1f706f96fb75882dec42e32a8471f18aefe6aea"} err="failed to get container status \"655c733148d2ec755cf22ea3d1f706f96fb75882dec42e32a8471f18aefe6aea\": rpc error: code = NotFound desc = could not find container \"655c733148d2ec755cf22ea3d1f706f96fb75882dec42e32a8471f18aefe6aea\": container with ID starting with 655c733148d2ec755cf22ea3d1f706f96fb75882dec42e32a8471f18aefe6aea not found: ID does not exist"
Sep 09 20:59:36.284516 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:36.284494    2556 scope.go:117] "RemoveContainer" containerID="1031d6af467c0e1e7bc728439f3aba764a56eed71cb9e2b85c503e014e800f88"
Sep 09 20:59:36.284737 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:59:36.284720    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1031d6af467c0e1e7bc728439f3aba764a56eed71cb9e2b85c503e014e800f88\": container with ID starting with 1031d6af467c0e1e7bc728439f3aba764a56eed71cb9e2b85c503e014e800f88 not found: ID does not exist" containerID="1031d6af467c0e1e7bc728439f3aba764a56eed71cb9e2b85c503e014e800f88"
Sep 09 20:59:36.284784 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:36.284742    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1031d6af467c0e1e7bc728439f3aba764a56eed71cb9e2b85c503e014e800f88"} err="failed to get container status \"1031d6af467c0e1e7bc728439f3aba764a56eed71cb9e2b85c503e014e800f88\": rpc error: code = NotFound desc = could not find container \"1031d6af467c0e1e7bc728439f3aba764a56eed71cb9e2b85c503e014e800f88\": container with ID starting with 1031d6af467c0e1e7bc728439f3aba764a56eed71cb9e2b85c503e014e800f88 not found: ID does not exist"
Sep 09 20:59:36.288468 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:36.288439    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"]
Sep 09 20:59:36.293156 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:36.293135    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ada93f-openshift-default-6ccc575b5b-64l7s"]
Sep 09 20:59:36.893730 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:36.893702    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5275b14-3ea3-45a3-b363-b203dc340646" path="/var/lib/kubelet/pods/c5275b14-3ea3-45a3-b363-b203dc340646/volumes"
Sep 09 20:59:38.727125 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:38.727089    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b2343-lnlz9"]
Sep 09 20:59:38.727642 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:38.727587    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5275b14-3ea3-45a3-b363-b203dc340646" containerName="istio-proxy"
Sep 09 20:59:38.727642 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:38.727601    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5275b14-3ea3-45a3-b363-b203dc340646" containerName="istio-proxy"
Sep 09 20:59:38.727642 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:38.727619    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b8a51ec-9ef7-4797-9b50-2b0abed94830" containerName="revoke-keys"
Sep 09 20:59:38.727642 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:38.727625    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b8a51ec-9ef7-4797-9b50-2b0abed94830" containerName="revoke-keys"
Sep 09 20:59:38.727882 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:38.727699    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5275b14-3ea3-45a3-b363-b203dc340646" containerName="istio-proxy"
Sep 09 20:59:38.727882 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:38.727713    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b8a51ec-9ef7-4797-9b50-2b0abed94830" containerName="revoke-keys"
Sep 09 20:59:38.732501 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:38.732479    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b2343-lnlz9"
Sep 09 20:59:38.735228 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:38.735207    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsks9\""
Sep 09 20:59:38.740220 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:38.740192    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b2343-lnlz9"]
Sep 09 20:59:38.813401 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:38.813366    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c231e92e-e877-4533-b0bd-cc9fbebce157-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2b2343-lnlz9\" (UID: \"c231e92e-e877-4533-b0bd-cc9fbebce157\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b2343-lnlz9"
Sep 09 20:59:38.914503 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:38.914479    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c231e92e-e877-4533-b0bd-cc9fbebce157-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2b2343-lnlz9\" (UID: \"c231e92e-e877-4533-b0bd-cc9fbebce157\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b2343-lnlz9"
Sep 09 20:59:38.915498 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:38.915466    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c231e92e-e877-4533-b0bd-cc9fbebce157-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2b2343-lnlz9\" (UID: \"c231e92e-e877-4533-b0bd-cc9fbebce157\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b2343-lnlz9"
Sep 09 20:59:39.045279 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:39.045250    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b2343-lnlz9"
Sep 09 20:59:39.325461 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:39.325421    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b2343-lnlz9"]
Sep 09 20:59:39.326208 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:59:39.326188    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc231e92e_e877_4533_b0bd_cc9fbebce157.slice/crio-a5d53b71bfadeef9ddaabd79f115a78c39053d325219589827f434510a09395d WatchSource:0}: Error finding container a5d53b71bfadeef9ddaabd79f115a78c39053d325219589827f434510a09395d: Status 404 returned error can't find the container with id a5d53b71bfadeef9ddaabd79f115a78c39053d325219589827f434510a09395d
Sep 09 20:59:40.298150 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:40.298117    2556 generic.go:356] "Generic (PLEG): container finished" podID="c231e92e-e877-4533-b0bd-cc9fbebce157" containerID="0c24c98e42e88a97c9fa32958731d2709d69b638009a1824165a9a95980f118d" exitCode=0
Sep 09 20:59:40.298514 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:40.298201    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b2343-lnlz9" event={"ID":"c231e92e-e877-4533-b0bd-cc9fbebce157","Type":"ContainerDied","Data":"0c24c98e42e88a97c9fa32958731d2709d69b638009a1824165a9a95980f118d"}
Sep 09 20:59:40.298514 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:40.298257    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b2343-lnlz9" event={"ID":"c231e92e-e877-4533-b0bd-cc9fbebce157","Type":"ContainerStarted","Data":"a5d53b71bfadeef9ddaabd79f115a78c39053d325219589827f434510a09395d"}
Sep 09 20:59:41.446379 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:41.446357    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b2343-lnlz9"
Sep 09 20:59:41.542441 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:41.542414    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c231e92e-e877-4533-b0bd-cc9fbebce157-maas-api-service-ca\") pod \"c231e92e-e877-4533-b0bd-cc9fbebce157\" (UID: \"c231e92e-e877-4533-b0bd-cc9fbebce157\") "
Sep 09 20:59:41.542771 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:41.542746    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c231e92e-e877-4533-b0bd-cc9fbebce157-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c231e92e-e877-4533-b0bd-cc9fbebce157" (UID: "c231e92e-e877-4533-b0bd-cc9fbebce157"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:59:41.643962 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:41.643867    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c231e92e-e877-4533-b0bd-cc9fbebce157-maas-api-service-ca\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:42.315206 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:42.312376    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b2343-lnlz9" event={"ID":"c231e92e-e877-4533-b0bd-cc9fbebce157","Type":"ContainerDied","Data":"a5d53b71bfadeef9ddaabd79f115a78c39053d325219589827f434510a09395d"}
Sep 09 20:59:42.315206 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:42.312518    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5d53b71bfadeef9ddaabd79f115a78c39053d325219589827f434510a09395d"
Sep 09 20:59:42.315206 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:42.312583    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b2343-lnlz9"
Sep 09 20:59:43.609806 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:43.609767    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"]
Sep 09 20:59:43.650001 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:43.649973    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"]
Sep 09 20:59:46.849327 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:46.849290    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw"]
Sep 09 20:59:46.849822 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:46.849513    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw" podUID="3d4a2d20-44d6-438b-8471-722d4d0e3b81" containerName="payload-processing" containerID="cri-o://84ba98455d34bb7ab1d0333cc9c5a0370c8fdb249897c73f5cd59a097f32d2ef" gracePeriod=30
Sep 09 20:59:47.050511 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:47.050475    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e2f93-5nvfx"]
Sep 09 20:59:47.055416 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:47.055391    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e2f93-5nvfx"]
Sep 09 20:59:47.111559 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:47.111496    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw"
Sep 09 20:59:47.190992 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:47.190963    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4v7m5\" (UniqueName: \"kubernetes.io/projected/3d4a2d20-44d6-438b-8471-722d4d0e3b81-kube-api-access-4v7m5\") pod \"3d4a2d20-44d6-438b-8471-722d4d0e3b81\" (UID: \"3d4a2d20-44d6-438b-8471-722d4d0e3b81\") "
Sep 09 20:59:47.191180 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:47.191057    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d4a2d20-44d6-438b-8471-722d4d0e3b81-plugins-config-volume\") pod \"3d4a2d20-44d6-438b-8471-722d4d0e3b81\" (UID: \"3d4a2d20-44d6-438b-8471-722d4d0e3b81\") "
Sep 09 20:59:47.191476 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:47.191450    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d4a2d20-44d6-438b-8471-722d4d0e3b81-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d4a2d20-44d6-438b-8471-722d4d0e3b81" (UID: "3d4a2d20-44d6-438b-8471-722d4d0e3b81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:59:47.193256 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:47.193227    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4a2d20-44d6-438b-8471-722d4d0e3b81-kube-api-access-4v7m5" (OuterVolumeSpecName: "kube-api-access-4v7m5") pod "3d4a2d20-44d6-438b-8471-722d4d0e3b81" (UID: "3d4a2d20-44d6-438b-8471-722d4d0e3b81"). InnerVolumeSpecName "kube-api-access-4v7m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:59:47.292809 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:47.292778    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v7m5\" (UniqueName: \"kubernetes.io/projected/3d4a2d20-44d6-438b-8471-722d4d0e3b81-kube-api-access-4v7m5\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:47.292809 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:47.292805    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d4a2d20-44d6-438b-8471-722d4d0e3b81-plugins-config-volume\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:47.341430 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:47.341403    2556 generic.go:356] "Generic (PLEG): container finished" podID="3d4a2d20-44d6-438b-8471-722d4d0e3b81" containerID="84ba98455d34bb7ab1d0333cc9c5a0370c8fdb249897c73f5cd59a097f32d2ef" exitCode=0
Sep 09 20:59:47.341554 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:47.341456    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw"
Sep 09 20:59:47.341554 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:47.341493    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw" event={"ID":"3d4a2d20-44d6-438b-8471-722d4d0e3b81","Type":"ContainerDied","Data":"84ba98455d34bb7ab1d0333cc9c5a0370c8fdb249897c73f5cd59a097f32d2ef"}
Sep 09 20:59:47.341554 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:47.341548    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw" event={"ID":"3d4a2d20-44d6-438b-8471-722d4d0e3b81","Type":"ContainerDied","Data":"12e22397aae15bcd4b010ce8cbf02a1f1d96ec7e738acaca77c6c8c484be5959"}
Sep 09 20:59:47.341673 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:47.341575    2556 scope.go:117] "RemoveContainer" containerID="84ba98455d34bb7ab1d0333cc9c5a0370c8fdb249897c73f5cd59a097f32d2ef"
Sep 09 20:59:47.351451 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:47.351434    2556 scope.go:117] "RemoveContainer" containerID="84ba98455d34bb7ab1d0333cc9c5a0370c8fdb249897c73f5cd59a097f32d2ef"
Sep 09 20:59:47.351717 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:59:47.351698    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ba98455d34bb7ab1d0333cc9c5a0370c8fdb249897c73f5cd59a097f32d2ef\": container with ID starting with 84ba98455d34bb7ab1d0333cc9c5a0370c8fdb249897c73f5cd59a097f32d2ef not found: ID does not exist" containerID="84ba98455d34bb7ab1d0333cc9c5a0370c8fdb249897c73f5cd59a097f32d2ef"
Sep 09 20:59:47.351762 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:47.351727    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ba98455d34bb7ab1d0333cc9c5a0370c8fdb249897c73f5cd59a097f32d2ef"} err="failed to get container status \"84ba98455d34bb7ab1d0333cc9c5a0370c8fdb249897c73f5cd59a097f32d2ef\": rpc error: code = NotFound desc = could not find container \"84ba98455d34bb7ab1d0333cc9c5a0370c8fdb249897c73f5cd59a097f32d2ef\": container with ID starting with 84ba98455d34bb7ab1d0333cc9c5a0370c8fdb249897c73f5cd59a097f32d2ef not found: ID does not exist"
Sep 09 20:59:47.365547 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:47.365494    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw"]
Sep 09 20:59:47.369277 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:47.369257    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4e2f93-69d5dd988c-lgxdw"]
Sep 09 20:59:48.893104 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:48.893070    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d4a2d20-44d6-438b-8471-722d4d0e3b81" path="/var/lib/kubelet/pods/3d4a2d20-44d6-438b-8471-722d4d0e3b81/volumes"
Sep 09 20:59:48.893474 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:48.893427    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b8a51ec-9ef7-4797-9b50-2b0abed94830" path="/var/lib/kubelet/pods/5b8a51ec-9ef7-4797-9b50-2b0abed94830/volumes"
Sep 09 20:59:54.038740 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:54.038705    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-78a7d5-lg2hp"]
Sep 09 20:59:54.039385 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:54.039364    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5275b14-3ea3-45a3-b363-b203dc340646" containerName="istio-proxy"
Sep 09 20:59:54.039487 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:54.039388    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5275b14-3ea3-45a3-b363-b203dc340646" containerName="istio-proxy"
Sep 09 20:59:54.039487 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:54.039400    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c231e92e-e877-4533-b0bd-cc9fbebce157" containerName="revoke-keys"
Sep 09 20:59:54.039487 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:54.039408    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c231e92e-e877-4533-b0bd-cc9fbebce157" containerName="revoke-keys"
Sep 09 20:59:54.039487 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:54.039424    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d4a2d20-44d6-438b-8471-722d4d0e3b81" containerName="payload-processing"
Sep 09 20:59:54.039487 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:54.039432    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d4a2d20-44d6-438b-8471-722d4d0e3b81" containerName="payload-processing"
Sep 09 20:59:54.039730 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:54.039571    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c231e92e-e877-4533-b0bd-cc9fbebce157" containerName="revoke-keys"
Sep 09 20:59:54.039730 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:54.039584    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5275b14-3ea3-45a3-b363-b203dc340646" containerName="istio-proxy"
Sep 09 20:59:54.039730 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:54.039596    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d4a2d20-44d6-438b-8471-722d4d0e3b81" containerName="payload-processing"
Sep 09 20:59:54.044115 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:54.044094    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-78a7d5-lg2hp"
Sep 09 20:59:54.046869 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:54.046848    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsks9\""
Sep 09 20:59:54.051417 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:54.051393    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-78a7d5-lg2hp"]
Sep 09 20:59:54.155218 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:54.155186    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea738aa3-9847-4681-9e2c-5b64f39a58b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-78a7d5-lg2hp\" (UID: \"ea738aa3-9847-4681-9e2c-5b64f39a58b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-78a7d5-lg2hp"
Sep 09 20:59:54.256366 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:54.256331    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea738aa3-9847-4681-9e2c-5b64f39a58b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-78a7d5-lg2hp\" (UID: \"ea738aa3-9847-4681-9e2c-5b64f39a58b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-78a7d5-lg2hp"
Sep 09 20:59:54.257027 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:54.257006    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea738aa3-9847-4681-9e2c-5b64f39a58b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-78a7d5-lg2hp\" (UID: \"ea738aa3-9847-4681-9e2c-5b64f39a58b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-78a7d5-lg2hp"
Sep 09 20:59:54.365261 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:54.365188    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-78a7d5-lg2hp"
Sep 09 20:59:54.640814 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:54.640786    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-78a7d5-lg2hp"]
Sep 09 20:59:54.642406 ip-10-0-135-117 kubenswrapper[2556]: W0909 20:59:54.642376    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea738aa3_9847_4681_9e2c_5b64f39a58b6.slice/crio-de68e84b03c1954b1f9df51f99f1ba640898fe9be60921a1aad454b2421204ab WatchSource:0}: Error finding container de68e84b03c1954b1f9df51f99f1ba640898fe9be60921a1aad454b2421204ab: Status 404 returned error can't find the container with id de68e84b03c1954b1f9df51f99f1ba640898fe9be60921a1aad454b2421204ab
Sep 09 20:59:55.391476 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.391446    2556 generic.go:356] "Generic (PLEG): container finished" podID="ea738aa3-9847-4681-9e2c-5b64f39a58b6" containerID="779e7751dc1c515878f09076200959441b5fe1e5e17178292d4e77f65ea86ed5" exitCode=0
Sep 09 20:59:55.391783 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.391514    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-78a7d5-lg2hp" event={"ID":"ea738aa3-9847-4681-9e2c-5b64f39a58b6","Type":"ContainerDied","Data":"779e7751dc1c515878f09076200959441b5fe1e5e17178292d4e77f65ea86ed5"}
Sep 09 20:59:55.391783 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.391550    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-78a7d5-lg2hp" event={"ID":"ea738aa3-9847-4681-9e2c-5b64f39a58b6","Type":"ContainerStarted","Data":"de68e84b03c1954b1f9df51f99f1ba640898fe9be60921a1aad454b2421204ab"}
Sep 09 20:59:55.464351 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.464319    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"]
Sep 09 20:59:55.464588 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.464560    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq" podUID="7f8c8e8a-d448-4a16-ab85-50518af36fcf" containerName="maas-api" containerID="cri-o://345ef4467df4892b97d4ba3a67753319e51da8a695a73a6c62baace5cac9915a" gracePeriod=30
Sep 09 20:59:55.535775 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.535716    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9"]
Sep 09 20:59:55.535947 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.535897    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9" podUID="f3c1d67e-0033-4ad7-8c03-58e7fc069e0a" containerName="payload-processing" containerID="cri-o://63171f67cce892ae29c61d87a8930683df1d5e1db2f2a8e01433e8846d423ca0" gracePeriod=30
Sep 09 20:59:55.730037 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.729191    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b2343-lnlz9"]
Sep 09 20:59:55.732871 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.732843    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b2343-lnlz9"]
Sep 09 20:59:55.791456 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.791430    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"
Sep 09 20:59:55.810838 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.809314    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9"
Sep 09 20:59:55.874078 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.874044    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8xzx\" (UniqueName: \"kubernetes.io/projected/f3c1d67e-0033-4ad7-8c03-58e7fc069e0a-kube-api-access-m8xzx\") pod \"f3c1d67e-0033-4ad7-8c03-58e7fc069e0a\" (UID: \"f3c1d67e-0033-4ad7-8c03-58e7fc069e0a\") "
Sep 09 20:59:55.874254 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.874111    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f8c8e8a-d448-4a16-ab85-50518af36fcf-metrics-certs\") pod \"7f8c8e8a-d448-4a16-ab85-50518af36fcf\" (UID: \"7f8c8e8a-d448-4a16-ab85-50518af36fcf\") "
Sep 09 20:59:55.874254 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.874174    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f8c8e8a-d448-4a16-ab85-50518af36fcf-openshift-service-ca\") pod \"7f8c8e8a-d448-4a16-ab85-50518af36fcf\" (UID: \"7f8c8e8a-d448-4a16-ab85-50518af36fcf\") "
Sep 09 20:59:55.874356 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.874258    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnhft\" (UniqueName: \"kubernetes.io/projected/7f8c8e8a-d448-4a16-ab85-50518af36fcf-kube-api-access-pnhft\") pod \"7f8c8e8a-d448-4a16-ab85-50518af36fcf\" (UID: \"7f8c8e8a-d448-4a16-ab85-50518af36fcf\") "
Sep 09 20:59:55.874356 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.874325    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f8c8e8a-d448-4a16-ab85-50518af36fcf-maas-api-tls\") pod \"7f8c8e8a-d448-4a16-ab85-50518af36fcf\" (UID: \"7f8c8e8a-d448-4a16-ab85-50518af36fcf\") "
Sep 09 20:59:55.874454 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.874384    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c1d67e-0033-4ad7-8c03-58e7fc069e0a-plugins-config-volume\") pod \"f3c1d67e-0033-4ad7-8c03-58e7fc069e0a\" (UID: \"f3c1d67e-0033-4ad7-8c03-58e7fc069e0a\") "
Sep 09 20:59:55.874606 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.874552    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f8c8e8a-d448-4a16-ab85-50518af36fcf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7f8c8e8a-d448-4a16-ab85-50518af36fcf" (UID: "7f8c8e8a-d448-4a16-ab85-50518af36fcf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:59:55.874874 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.874843    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3c1d67e-0033-4ad7-8c03-58e7fc069e0a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f3c1d67e-0033-4ad7-8c03-58e7fc069e0a" (UID: "f3c1d67e-0033-4ad7-8c03-58e7fc069e0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:59:55.874996 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.874889    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f8c8e8a-d448-4a16-ab85-50518af36fcf-openshift-service-ca\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:55.876808 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.876778    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f8c8e8a-d448-4a16-ab85-50518af36fcf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7f8c8e8a-d448-4a16-ab85-50518af36fcf" (UID: "7f8c8e8a-d448-4a16-ab85-50518af36fcf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:59:55.876884 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.876795    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8c8e8a-d448-4a16-ab85-50518af36fcf-kube-api-access-pnhft" (OuterVolumeSpecName: "kube-api-access-pnhft") pod "7f8c8e8a-d448-4a16-ab85-50518af36fcf" (UID: "7f8c8e8a-d448-4a16-ab85-50518af36fcf"). InnerVolumeSpecName "kube-api-access-pnhft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:59:55.876884 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.876802    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f8c8e8a-d448-4a16-ab85-50518af36fcf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7f8c8e8a-d448-4a16-ab85-50518af36fcf" (UID: "7f8c8e8a-d448-4a16-ab85-50518af36fcf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:59:55.877007 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.876987    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c1d67e-0033-4ad7-8c03-58e7fc069e0a-kube-api-access-m8xzx" (OuterVolumeSpecName: "kube-api-access-m8xzx") pod "f3c1d67e-0033-4ad7-8c03-58e7fc069e0a" (UID: "f3c1d67e-0033-4ad7-8c03-58e7fc069e0a"). InnerVolumeSpecName "kube-api-access-m8xzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:59:55.976114 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.976079    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnhft\" (UniqueName: \"kubernetes.io/projected/7f8c8e8a-d448-4a16-ab85-50518af36fcf-kube-api-access-pnhft\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:55.976114 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.976115    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f8c8e8a-d448-4a16-ab85-50518af36fcf-maas-api-tls\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:55.976305 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.976133    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c1d67e-0033-4ad7-8c03-58e7fc069e0a-plugins-config-volume\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:55.976305 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.976147    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8xzx\" (UniqueName: \"kubernetes.io/projected/f3c1d67e-0033-4ad7-8c03-58e7fc069e0a-kube-api-access-m8xzx\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:55.976305 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:55.976161    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f8c8e8a-d448-4a16-ab85-50518af36fcf-metrics-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:56.398086 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.398050    2556 generic.go:356] "Generic (PLEG): container finished" podID="7f8c8e8a-d448-4a16-ab85-50518af36fcf" containerID="345ef4467df4892b97d4ba3a67753319e51da8a695a73a6c62baace5cac9915a" exitCode=0
Sep 09 20:59:56.398533 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.398099    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"
Sep 09 20:59:56.398533 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.398137    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq" event={"ID":"7f8c8e8a-d448-4a16-ab85-50518af36fcf","Type":"ContainerDied","Data":"345ef4467df4892b97d4ba3a67753319e51da8a695a73a6c62baace5cac9915a"}
Sep 09 20:59:56.398533 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.398204    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq" event={"ID":"7f8c8e8a-d448-4a16-ab85-50518af36fcf","Type":"ContainerDied","Data":"ebb75e8ee7c0705bc39ad9af7ecb00cf229d70187b7070edcef228cdad2bd1ff"}
Sep 09 20:59:56.398533 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.398240    2556 scope.go:117] "RemoveContainer" containerID="345ef4467df4892b97d4ba3a67753319e51da8a695a73a6c62baace5cac9915a"
Sep 09 20:59:56.399588 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.399568    2556 generic.go:356] "Generic (PLEG): container finished" podID="f3c1d67e-0033-4ad7-8c03-58e7fc069e0a" containerID="63171f67cce892ae29c61d87a8930683df1d5e1db2f2a8e01433e8846d423ca0" exitCode=0
Sep 09 20:59:56.399656 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.399611    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9"
Sep 09 20:59:56.399656 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.399639    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9" event={"ID":"f3c1d67e-0033-4ad7-8c03-58e7fc069e0a","Type":"ContainerDied","Data":"63171f67cce892ae29c61d87a8930683df1d5e1db2f2a8e01433e8846d423ca0"}
Sep 09 20:59:56.399752 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.399687    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9" event={"ID":"f3c1d67e-0033-4ad7-8c03-58e7fc069e0a","Type":"ContainerDied","Data":"3a32a00a044e6d6e1b12770a53a2915707075439dc1ae192f3a8bc6c8d1c226b"}
Sep 09 20:59:56.408641 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.408621    2556 scope.go:117] "RemoveContainer" containerID="345ef4467df4892b97d4ba3a67753319e51da8a695a73a6c62baace5cac9915a"
Sep 09 20:59:56.408946 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:59:56.408895    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"345ef4467df4892b97d4ba3a67753319e51da8a695a73a6c62baace5cac9915a\": container with ID starting with 345ef4467df4892b97d4ba3a67753319e51da8a695a73a6c62baace5cac9915a not found: ID does not exist" containerID="345ef4467df4892b97d4ba3a67753319e51da8a695a73a6c62baace5cac9915a"
Sep 09 20:59:56.409031 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.408956    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"345ef4467df4892b97d4ba3a67753319e51da8a695a73a6c62baace5cac9915a"} err="failed to get container status \"345ef4467df4892b97d4ba3a67753319e51da8a695a73a6c62baace5cac9915a\": rpc error: code = NotFound desc = could not find container \"345ef4467df4892b97d4ba3a67753319e51da8a695a73a6c62baace5cac9915a\": container with ID starting with 345ef4467df4892b97d4ba3a67753319e51da8a695a73a6c62baace5cac9915a not found: ID does not exist"
Sep 09 20:59:56.409031 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.408975    2556 scope.go:117] "RemoveContainer" containerID="63171f67cce892ae29c61d87a8930683df1d5e1db2f2a8e01433e8846d423ca0"
Sep 09 20:59:56.418150 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.418130    2556 scope.go:117] "RemoveContainer" containerID="63171f67cce892ae29c61d87a8930683df1d5e1db2f2a8e01433e8846d423ca0"
Sep 09 20:59:56.418481 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:59:56.418458    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63171f67cce892ae29c61d87a8930683df1d5e1db2f2a8e01433e8846d423ca0\": container with ID starting with 63171f67cce892ae29c61d87a8930683df1d5e1db2f2a8e01433e8846d423ca0 not found: ID does not exist" containerID="63171f67cce892ae29c61d87a8930683df1d5e1db2f2a8e01433e8846d423ca0"
Sep 09 20:59:56.418563 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.418486    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63171f67cce892ae29c61d87a8930683df1d5e1db2f2a8e01433e8846d423ca0"} err="failed to get container status \"63171f67cce892ae29c61d87a8930683df1d5e1db2f2a8e01433e8846d423ca0\": rpc error: code = NotFound desc = could not find container \"63171f67cce892ae29c61d87a8930683df1d5e1db2f2a8e01433e8846d423ca0\": container with ID starting with 63171f67cce892ae29c61d87a8930683df1d5e1db2f2a8e01433e8846d423ca0 not found: ID does not exist"
Sep 09 20:59:56.426207 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.426183    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"]
Sep 09 20:59:56.431269 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.430783    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq"]
Sep 09 20:59:56.444162 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.444140    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9"]
Sep 09 20:59:56.448388 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.448364    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2b2343-58886c7587-888t9"]
Sep 09 20:59:56.538453 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.538431    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-78a7d5-lg2hp"
Sep 09 20:59:56.683008 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.682943    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea738aa3-9847-4681-9e2c-5b64f39a58b6-maas-api-service-ca\") pod \"ea738aa3-9847-4681-9e2c-5b64f39a58b6\" (UID: \"ea738aa3-9847-4681-9e2c-5b64f39a58b6\") "
Sep 09 20:59:56.683290 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.683268    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea738aa3-9847-4681-9e2c-5b64f39a58b6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ea738aa3-9847-4681-9e2c-5b64f39a58b6" (UID: "ea738aa3-9847-4681-9e2c-5b64f39a58b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:59:56.783597 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.783567    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea738aa3-9847-4681-9e2c-5b64f39a58b6-maas-api-service-ca\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:56.894014 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.893986    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f8c8e8a-d448-4a16-ab85-50518af36fcf" path="/var/lib/kubelet/pods/7f8c8e8a-d448-4a16-ab85-50518af36fcf/volumes"
Sep 09 20:59:56.894399 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.894384    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c231e92e-e877-4533-b0bd-cc9fbebce157" path="/var/lib/kubelet/pods/c231e92e-e877-4533-b0bd-cc9fbebce157/volumes"
Sep 09 20:59:56.894721 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:56.894709    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3c1d67e-0033-4ad7-8c03-58e7fc069e0a" path="/var/lib/kubelet/pods/f3c1d67e-0033-4ad7-8c03-58e7fc069e0a/volumes"
Sep 09 20:59:57.408939 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:57.408896    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-78a7d5-lg2hp"
Sep 09 20:59:57.409373 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:57.408941    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-78a7d5-lg2hp" event={"ID":"ea738aa3-9847-4681-9e2c-5b64f39a58b6","Type":"ContainerDied","Data":"de68e84b03c1954b1f9df51f99f1ba640898fe9be60921a1aad454b2421204ab"}
Sep 09 20:59:57.409373 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:57.409003    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de68e84b03c1954b1f9df51f99f1ba640898fe9be60921a1aad454b2421204ab"
Sep 09 20:59:58.630212 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:58.630164    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2" podUID="55619605-52ef-4109-8856-d64d81a6d8d4" containerName="main" containerID="cri-o://c95406e4e5276c8722c61def0c37c5c336493e70308e929d2738d551368b951d" gracePeriod=45
Sep 09 20:59:58.669167 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:58.669131    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8" podUID="ea37b5f6-441f-458c-a4ba-6949c906cbfa" containerName="main" containerID="cri-o://4e277ef37c992ab5fc21752270289d05df9e1cd967b42b0b75800977e575aa32" gracePeriod=45
Sep 09 20:59:58.719006 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:58.718973    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b2343-69ffbc96bd-sq7lq" podUID="7f8c8e8a-d448-4a16-ab85-50518af36fcf" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.34:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 09 20:59:58.935609 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:58.935577    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:59:58.952629 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:58.952608    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:59:59.007187 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.007162    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-tmp-dir\") pod \"55619605-52ef-4109-8856-d64d81a6d8d4\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") "
Sep 09 20:59:59.007357 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.007199    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99cdb\" (UniqueName: \"kubernetes.io/projected/55619605-52ef-4109-8856-d64d81a6d8d4-kube-api-access-99cdb\") pod \"55619605-52ef-4109-8856-d64d81a6d8d4\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") "
Sep 09 20:59:59.007357 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.007239    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-dshm\") pod \"55619605-52ef-4109-8856-d64d81a6d8d4\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") "
Sep 09 20:59:59.007357 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.007274    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-model-cache\") pod \"55619605-52ef-4109-8856-d64d81a6d8d4\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") "
Sep 09 20:59:59.007357 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.007291    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55619605-52ef-4109-8856-d64d81a6d8d4-tls-certs\") pod \"55619605-52ef-4109-8856-d64d81a6d8d4\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") "
Sep 09 20:59:59.007738 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.007359    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-home\") pod \"55619605-52ef-4109-8856-d64d81a6d8d4\" (UID: \"55619605-52ef-4109-8856-d64d81a6d8d4\") "
Sep 09 20:59:59.007738 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.007429    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55619605-52ef-4109-8856-d64d81a6d8d4" (UID: "55619605-52ef-4109-8856-d64d81a6d8d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:59:59.007738 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.007627    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "55619605-52ef-4109-8856-d64d81a6d8d4" (UID: "55619605-52ef-4109-8856-d64d81a6d8d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:59:59.008543 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.007734    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-home" (OuterVolumeSpecName: "home") pod "55619605-52ef-4109-8856-d64d81a6d8d4" (UID: "55619605-52ef-4109-8856-d64d81a6d8d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:59:59.008543 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.007939    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-home\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:59.008543 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.007960    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-tmp-dir\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:59.008543 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.007975    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-model-cache\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:59.009962 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.009938    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-dshm" (OuterVolumeSpecName: "dshm") pod "55619605-52ef-4109-8856-d64d81a6d8d4" (UID: "55619605-52ef-4109-8856-d64d81a6d8d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:59:59.010028 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.009938    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55619605-52ef-4109-8856-d64d81a6d8d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55619605-52ef-4109-8856-d64d81a6d8d4" (UID: "55619605-52ef-4109-8856-d64d81a6d8d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:59:59.010028 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.010006    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55619605-52ef-4109-8856-d64d81a6d8d4-kube-api-access-99cdb" (OuterVolumeSpecName: "kube-api-access-99cdb") pod "55619605-52ef-4109-8856-d64d81a6d8d4" (UID: "55619605-52ef-4109-8856-d64d81a6d8d4"). InnerVolumeSpecName "kube-api-access-99cdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:59:59.109156 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.109126    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-home\") pod \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") "
Sep 09 20:59:59.109301 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.109190    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea37b5f6-441f-458c-a4ba-6949c906cbfa-tls-certs\") pod \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") "
Sep 09 20:59:59.109301 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.109211    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-model-cache\") pod \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") "
Sep 09 20:59:59.109301 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.109270    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-dshm\") pod \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") "
Sep 09 20:59:59.109301 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.109291    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzpcc\" (UniqueName: \"kubernetes.io/projected/ea37b5f6-441f-458c-a4ba-6949c906cbfa-kube-api-access-tzpcc\") pod \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") "
Sep 09 20:59:59.109540 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.109362    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-tmp-dir\") pod \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\" (UID: \"ea37b5f6-441f-458c-a4ba-6949c906cbfa\") "
Sep 09 20:59:59.109540 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.109420    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-home" (OuterVolumeSpecName: "home") pod "ea37b5f6-441f-458c-a4ba-6949c906cbfa" (UID: "ea37b5f6-441f-458c-a4ba-6949c906cbfa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:59:59.109540 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.109514    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea37b5f6-441f-458c-a4ba-6949c906cbfa" (UID: "ea37b5f6-441f-458c-a4ba-6949c906cbfa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:59:59.109713 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.109623    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea37b5f6-441f-458c-a4ba-6949c906cbfa" (UID: "ea37b5f6-441f-458c-a4ba-6949c906cbfa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:59:59.109827 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.109801    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-home\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:59.109827 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.109827    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-model-cache\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:59.110037 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.109842    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99cdb\" (UniqueName: \"kubernetes.io/projected/55619605-52ef-4109-8856-d64d81a6d8d4-kube-api-access-99cdb\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:59.110037 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.109856    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55619605-52ef-4109-8856-d64d81a6d8d4-dshm\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:59.110037 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.109867    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55619605-52ef-4109-8856-d64d81a6d8d4-tls-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:59.110037 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.109879    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-tmp-dir\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:59.111553 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.111523    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea37b5f6-441f-458c-a4ba-6949c906cbfa-kube-api-access-tzpcc" (OuterVolumeSpecName: "kube-api-access-tzpcc") pod "ea37b5f6-441f-458c-a4ba-6949c906cbfa" (UID: "ea37b5f6-441f-458c-a4ba-6949c906cbfa"). InnerVolumeSpecName "kube-api-access-tzpcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:59:59.111657 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.111557    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-dshm" (OuterVolumeSpecName: "dshm") pod "ea37b5f6-441f-458c-a4ba-6949c906cbfa" (UID: "ea37b5f6-441f-458c-a4ba-6949c906cbfa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:59:59.111780 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.111760    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea37b5f6-441f-458c-a4ba-6949c906cbfa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea37b5f6-441f-458c-a4ba-6949c906cbfa" (UID: "ea37b5f6-441f-458c-a4ba-6949c906cbfa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:59:59.211713 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.211643    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea37b5f6-441f-458c-a4ba-6949c906cbfa-dshm\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:59.211713 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.211681    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzpcc\" (UniqueName: \"kubernetes.io/projected/ea37b5f6-441f-458c-a4ba-6949c906cbfa-kube-api-access-tzpcc\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:59.211713 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.211696    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea37b5f6-441f-458c-a4ba-6949c906cbfa-tls-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 20:59:59.424961 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.424930    2556 generic.go:356] "Generic (PLEG): container finished" podID="55619605-52ef-4109-8856-d64d81a6d8d4" containerID="c95406e4e5276c8722c61def0c37c5c336493e70308e929d2738d551368b951d" exitCode=0
Sep 09 20:59:59.425122 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.424965    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"
Sep 09 20:59:59.425122 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.425007    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2" event={"ID":"55619605-52ef-4109-8856-d64d81a6d8d4","Type":"ContainerDied","Data":"c95406e4e5276c8722c61def0c37c5c336493e70308e929d2738d551368b951d"}
Sep 09 20:59:59.425122 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.425059    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2" event={"ID":"55619605-52ef-4109-8856-d64d81a6d8d4","Type":"ContainerDied","Data":"cd34aede7d16efc79114a99aba69d69569a624437e05348b545979ce1689c93e"}
Sep 09 20:59:59.425122 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.425084    2556 scope.go:117] "RemoveContainer" containerID="c95406e4e5276c8722c61def0c37c5c336493e70308e929d2738d551368b951d"
Sep 09 20:59:59.426473 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.426448    2556 generic.go:356] "Generic (PLEG): container finished" podID="ea37b5f6-441f-458c-a4ba-6949c906cbfa" containerID="4e277ef37c992ab5fc21752270289d05df9e1cd967b42b0b75800977e575aa32" exitCode=0
Sep 09 20:59:59.426565 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.426477    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8" event={"ID":"ea37b5f6-441f-458c-a4ba-6949c906cbfa","Type":"ContainerDied","Data":"4e277ef37c992ab5fc21752270289d05df9e1cd967b42b0b75800977e575aa32"}
Sep 09 20:59:59.426565 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.426521    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8" event={"ID":"ea37b5f6-441f-458c-a4ba-6949c906cbfa","Type":"ContainerDied","Data":"02444f0b240ca0165c02031f6e97231afbc686662b1b4948009149b4eab519c2"}
Sep 09 20:59:59.426565 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.426493    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"
Sep 09 20:59:59.434750 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.434732    2556 scope.go:117] "RemoveContainer" containerID="c95406e4e5276c8722c61def0c37c5c336493e70308e929d2738d551368b951d"
Sep 09 20:59:59.435053 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:59:59.435033    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c95406e4e5276c8722c61def0c37c5c336493e70308e929d2738d551368b951d\": container with ID starting with c95406e4e5276c8722c61def0c37c5c336493e70308e929d2738d551368b951d not found: ID does not exist" containerID="c95406e4e5276c8722c61def0c37c5c336493e70308e929d2738d551368b951d"
Sep 09 20:59:59.435109 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.435075    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c95406e4e5276c8722c61def0c37c5c336493e70308e929d2738d551368b951d"} err="failed to get container status \"c95406e4e5276c8722c61def0c37c5c336493e70308e929d2738d551368b951d\": rpc error: code = NotFound desc = could not find container \"c95406e4e5276c8722c61def0c37c5c336493e70308e929d2738d551368b951d\": container with ID starting with c95406e4e5276c8722c61def0c37c5c336493e70308e929d2738d551368b951d not found: ID does not exist"
Sep 09 20:59:59.435109 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.435100    2556 scope.go:117] "RemoveContainer" containerID="4e277ef37c992ab5fc21752270289d05df9e1cd967b42b0b75800977e575aa32"
Sep 09 20:59:59.443330 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.443314    2556 scope.go:117] "RemoveContainer" containerID="4e277ef37c992ab5fc21752270289d05df9e1cd967b42b0b75800977e575aa32"
Sep 09 20:59:59.443571 ip-10-0-135-117 kubenswrapper[2556]: E0909 20:59:59.443555    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e277ef37c992ab5fc21752270289d05df9e1cd967b42b0b75800977e575aa32\": container with ID starting with 4e277ef37c992ab5fc21752270289d05df9e1cd967b42b0b75800977e575aa32 not found: ID does not exist" containerID="4e277ef37c992ab5fc21752270289d05df9e1cd967b42b0b75800977e575aa32"
Sep 09 20:59:59.443616 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.443579    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e277ef37c992ab5fc21752270289d05df9e1cd967b42b0b75800977e575aa32"} err="failed to get container status \"4e277ef37c992ab5fc21752270289d05df9e1cd967b42b0b75800977e575aa32\": rpc error: code = NotFound desc = could not find container \"4e277ef37c992ab5fc21752270289d05df9e1cd967b42b0b75800977e575aa32\": container with ID starting with 4e277ef37c992ab5fc21752270289d05df9e1cd967b42b0b75800977e575aa32 not found: ID does not exist"
Sep 09 20:59:59.451406 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.451386    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"]
Sep 09 20:59:59.455943 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.455924    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2b2343/distinct-w0-2b2343-kserve-f7f7f778f-s72g2"]
Sep 09 20:59:59.466189 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.466141    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"]
Sep 09 20:59:59.471744 ip-10-0-135-117 kubenswrapper[2556]: I0909 20:59:59.471715    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2b2343/unconfigured-w0-2b2343-kserve-f8c5bbc54-498f8"]
Sep 09 21:00:00.139404 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.139371    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fa05f-29816460-b6p2v"]
Sep 09 21:00:00.140024 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.140004    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea738aa3-9847-4681-9e2c-5b64f39a58b6" containerName="revoke-keys"
Sep 09 21:00:00.140084 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.140026    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea738aa3-9847-4681-9e2c-5b64f39a58b6" containerName="revoke-keys"
Sep 09 21:00:00.140084 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.140055    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c1d67e-0033-4ad7-8c03-58e7fc069e0a" containerName="payload-processing"
Sep 09 21:00:00.140084 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.140064    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c1d67e-0033-4ad7-8c03-58e7fc069e0a" containerName="payload-processing"
Sep 09 21:00:00.140084 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.140078    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55619605-52ef-4109-8856-d64d81a6d8d4" containerName="main"
Sep 09 21:00:00.140213 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.140086    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="55619605-52ef-4109-8856-d64d81a6d8d4" containerName="main"
Sep 09 21:00:00.140213 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.140101    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea37b5f6-441f-458c-a4ba-6949c906cbfa" containerName="main"
Sep 09 21:00:00.140213 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.140109    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea37b5f6-441f-458c-a4ba-6949c906cbfa" containerName="main"
Sep 09 21:00:00.140213 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.140134    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f8c8e8a-d448-4a16-ab85-50518af36fcf" containerName="maas-api"
Sep 09 21:00:00.140213 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.140142    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f8c8e8a-d448-4a16-ab85-50518af36fcf" containerName="maas-api"
Sep 09 21:00:00.140353 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.140256    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea738aa3-9847-4681-9e2c-5b64f39a58b6" containerName="revoke-keys"
Sep 09 21:00:00.140353 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.140271    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f8c8e8a-d448-4a16-ab85-50518af36fcf" containerName="maas-api"
Sep 09 21:00:00.140353 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.140280    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea37b5f6-441f-458c-a4ba-6949c906cbfa" containerName="main"
Sep 09 21:00:00.140353 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.140290    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="55619605-52ef-4109-8856-d64d81a6d8d4" containerName="main"
Sep 09 21:00:00.140353 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.140301    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c1d67e-0033-4ad7-8c03-58e7fc069e0a" containerName="payload-processing"
Sep 09 21:00:00.145453 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.145430    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-78a7d5-29816460-lxchn"]
Sep 09 21:00:00.145602 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.145563    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fa05f-29816460-b6p2v"
Sep 09 21:00:00.148600 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.148579    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsks9\""
Sep 09 21:00:00.149160 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.149140    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-933e9d-29816460-kjcxg"]
Sep 09 21:00:00.149265 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.149251    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-78a7d5-29816460-lxchn"
Sep 09 21:00:00.154342 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.154314    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cb82f2-29816460-m4knn"]
Sep 09 21:00:00.156176 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.154650    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-933e9d-29816460-kjcxg"
Sep 09 21:00:00.162610 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.161037    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cb82f2-29816460-m4knn"
Sep 09 21:00:00.162610 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.160965    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-933e9d-29816460-kjcxg"]
Sep 09 21:00:00.162610 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.161231    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-78a7d5-29816460-lxchn"]
Sep 09 21:00:00.162610 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.161260    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fa05f-29816460-b6p2v"]
Sep 09 21:00:00.163546 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.163416    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fa05f-29816460-b6p2v"
Sep 09 21:00:00.164488 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.164465    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cb82f2-29816460-m4knn"]
Sep 09 21:00:00.176859 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.176838    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-78a7d5-29816460-lxchn"
Sep 09 21:00:00.214052 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.214028    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-933e9d-29816460-kjcxg"
Sep 09 21:00:00.222246 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.222209    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cb82f2-29816460-m4knn"
Sep 09 21:00:00.458554 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.458528    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fa05f-29816460-b6p2v"]
Sep 09 21:00:00.459885 ip-10-0-135-117 kubenswrapper[2556]: W0909 21:00:00.459860    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8539a8e_cacb_4201_9397_300fbb23edec.slice/crio-8209e2495fad091df2f3d48e9d3d5e0b2e21d06d89356d5255d82ae04861fa49 WatchSource:0}: Error finding container 8209e2495fad091df2f3d48e9d3d5e0b2e21d06d89356d5255d82ae04861fa49: Status 404 returned error can't find the container with id 8209e2495fad091df2f3d48e9d3d5e0b2e21d06d89356d5255d82ae04861fa49
Sep 09 21:00:00.477080 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.476680    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-78a7d5-29816460-lxchn"]
Sep 09 21:00:00.477888 ip-10-0-135-117 kubenswrapper[2556]: W0909 21:00:00.477864    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod237bade6_8f58_45a4_8aa6_bdc9a3d03ed4.slice/crio-5c4ad760007a90b84c35cbf7eb7490ce7a05915a5cdc251b6b4307a7a7a0b453 WatchSource:0}: Error finding container 5c4ad760007a90b84c35cbf7eb7490ce7a05915a5cdc251b6b4307a7a7a0b453: Status 404 returned error can't find the container with id 5c4ad760007a90b84c35cbf7eb7490ce7a05915a5cdc251b6b4307a7a7a0b453
Sep 09 21:00:00.519573 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.519549    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-933e9d-29816460-kjcxg"]
Sep 09 21:00:00.520864 ip-10-0-135-117 kubenswrapper[2556]: W0909 21:00:00.520842    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a66023_786a_4984_914e_a15cb5fc394c.slice/crio-ef00474a857ea37092d936206a33644937d960f21ed2b3c62a9366e539a23f2f WatchSource:0}: Error finding container ef00474a857ea37092d936206a33644937d960f21ed2b3c62a9366e539a23f2f: Status 404 returned error can't find the container with id ef00474a857ea37092d936206a33644937d960f21ed2b3c62a9366e539a23f2f
Sep 09 21:00:00.532400 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.532377    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cb82f2-29816460-m4knn"]
Sep 09 21:00:00.534272 ip-10-0-135-117 kubenswrapper[2556]: W0909 21:00:00.534252    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105b2c9d_1d2c_49a5_b97e_7bb11473a158.slice/crio-90c6ef2166c45429576262b11f56cbe33bfbeaadac0ddd2caa141eb92c643f00 WatchSource:0}: Error finding container 90c6ef2166c45429576262b11f56cbe33bfbeaadac0ddd2caa141eb92c643f00: Status 404 returned error can't find the container with id 90c6ef2166c45429576262b11f56cbe33bfbeaadac0ddd2caa141eb92c643f00
Sep 09 21:00:00.893277 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.893251    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55619605-52ef-4109-8856-d64d81a6d8d4" path="/var/lib/kubelet/pods/55619605-52ef-4109-8856-d64d81a6d8d4/volumes"
Sep 09 21:00:00.893651 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:00.893638    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea37b5f6-441f-458c-a4ba-6949c906cbfa" path="/var/lib/kubelet/pods/ea37b5f6-441f-458c-a4ba-6949c906cbfa/volumes"
Sep 09 21:00:01.442422 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:01.442389    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-78a7d5-29816460-lxchn" event={"ID":"237bade6-8f58-45a4-8aa6-bdc9a3d03ed4","Type":"ContainerStarted","Data":"93d1933f71b582e88eddae12bef1dca40a6b57e247f05d79a63d967daa21c824"}
Sep 09 21:00:01.442829 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:01.442475    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-78a7d5-29816460-lxchn" event={"ID":"237bade6-8f58-45a4-8aa6-bdc9a3d03ed4","Type":"ContainerStarted","Data":"5c4ad760007a90b84c35cbf7eb7490ce7a05915a5cdc251b6b4307a7a7a0b453"}
Sep 09 21:00:01.443994 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:01.443973    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-933e9d-29816460-kjcxg" event={"ID":"c1a66023-786a-4984-914e-a15cb5fc394c","Type":"ContainerStarted","Data":"918d02628340a39b375280f4544eef39ccd8eafe67e20de4eb61ecdb260e92ee"}
Sep 09 21:00:01.444062 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:01.444022    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-933e9d-29816460-kjcxg" event={"ID":"c1a66023-786a-4984-914e-a15cb5fc394c","Type":"ContainerStarted","Data":"ef00474a857ea37092d936206a33644937d960f21ed2b3c62a9366e539a23f2f"}
Sep 09 21:00:01.445543 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:01.445511    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cb82f2-29816460-m4knn" event={"ID":"105b2c9d-1d2c-49a5-b97e-7bb11473a158","Type":"ContainerStarted","Data":"37b5fbcdab45fcf7555b30874971130b00104c38ebeb22b7e2de20677db6288f"}
Sep 09 21:00:01.445796 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:01.445562    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cb82f2-29816460-m4knn" event={"ID":"105b2c9d-1d2c-49a5-b97e-7bb11473a158","Type":"ContainerStarted","Data":"90c6ef2166c45429576262b11f56cbe33bfbeaadac0ddd2caa141eb92c643f00"}
Sep 09 21:00:01.446954 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:01.446933    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fa05f-29816460-b6p2v" event={"ID":"a8539a8e-cacb-4201-9397-300fbb23edec","Type":"ContainerStarted","Data":"00a7c013076210d547d17d73a1f50131547df48965fc09f67fee59dfac916de0"}
Sep 09 21:00:01.447084 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:01.446971    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fa05f-29816460-b6p2v" event={"ID":"a8539a8e-cacb-4201-9397-300fbb23edec","Type":"ContainerStarted","Data":"8209e2495fad091df2f3d48e9d3d5e0b2e21d06d89356d5255d82ae04861fa49"}
Sep 09 21:00:01.458894 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:01.458852    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-78a7d5-29816460-lxchn" podStartSLOduration=1.4588392 podStartE2EDuration="1.4588392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:01.458352921 +0000 UTC m=+1647.138313232" watchObservedRunningTime="2026-09-09 21:00:01.4588392 +0000 UTC m=+1647.138799510"
Sep 09 21:00:01.477493 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:01.477451    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fa05f-29816460-b6p2v" podStartSLOduration=1.477439361 podStartE2EDuration="1.477439361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:01.475919374 +0000 UTC m=+1647.155879678" watchObservedRunningTime="2026-09-09 21:00:01.477439361 +0000 UTC m=+1647.157399671"
Sep 09 21:00:01.493702 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:01.493661    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cb82f2-29816460-m4knn" podStartSLOduration=1.493651805 podStartE2EDuration="1.493651805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:01.491315059 +0000 UTC m=+1647.171275369" watchObservedRunningTime="2026-09-09 21:00:01.493651805 +0000 UTC m=+1647.173612111"
Sep 09 21:00:01.506357 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:01.506319    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-933e9d-29816460-kjcxg" podStartSLOduration=1.506307959 podStartE2EDuration="1.506307959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:01.506222235 +0000 UTC m=+1647.186182544" watchObservedRunningTime="2026-09-09 21:00:01.506307959 +0000 UTC m=+1647.186268268"
Sep 09 21:00:02.454867 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:02.454833    2556 generic.go:356] "Generic (PLEG): container finished" podID="237bade6-8f58-45a4-8aa6-bdc9a3d03ed4" containerID="93d1933f71b582e88eddae12bef1dca40a6b57e247f05d79a63d967daa21c824" exitCode=0
Sep 09 21:00:02.455333 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:02.454939    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-78a7d5-29816460-lxchn" event={"ID":"237bade6-8f58-45a4-8aa6-bdc9a3d03ed4","Type":"ContainerDied","Data":"93d1933f71b582e88eddae12bef1dca40a6b57e247f05d79a63d967daa21c824"}
Sep 09 21:00:02.456545 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:02.456523    2556 generic.go:356] "Generic (PLEG): container finished" podID="c1a66023-786a-4984-914e-a15cb5fc394c" containerID="918d02628340a39b375280f4544eef39ccd8eafe67e20de4eb61ecdb260e92ee" exitCode=0
Sep 09 21:00:02.456681 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:02.456597    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-933e9d-29816460-kjcxg" event={"ID":"c1a66023-786a-4984-914e-a15cb5fc394c","Type":"ContainerDied","Data":"918d02628340a39b375280f4544eef39ccd8eafe67e20de4eb61ecdb260e92ee"}
Sep 09 21:00:02.457992 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:02.457973    2556 generic.go:356] "Generic (PLEG): container finished" podID="105b2c9d-1d2c-49a5-b97e-7bb11473a158" containerID="37b5fbcdab45fcf7555b30874971130b00104c38ebeb22b7e2de20677db6288f" exitCode=0
Sep 09 21:00:02.458072 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:02.457997    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cb82f2-29816460-m4knn" event={"ID":"105b2c9d-1d2c-49a5-b97e-7bb11473a158","Type":"ContainerDied","Data":"37b5fbcdab45fcf7555b30874971130b00104c38ebeb22b7e2de20677db6288f"}
Sep 09 21:00:02.459535 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:02.459517    2556 generic.go:356] "Generic (PLEG): container finished" podID="a8539a8e-cacb-4201-9397-300fbb23edec" containerID="00a7c013076210d547d17d73a1f50131547df48965fc09f67fee59dfac916de0" exitCode=0
Sep 09 21:00:02.459613 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:02.459585    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fa05f-29816460-b6p2v" event={"ID":"a8539a8e-cacb-4201-9397-300fbb23edec","Type":"ContainerDied","Data":"00a7c013076210d547d17d73a1f50131547df48965fc09f67fee59dfac916de0"}
Sep 09 21:00:03.639659 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:03.639634    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-933e9d-29816460-kjcxg"
Sep 09 21:00:03.673505 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:03.673480    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fa05f-29816460-b6p2v"
Sep 09 21:00:03.707722 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:03.707702    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cb82f2-29816460-m4knn"
Sep 09 21:00:03.711373 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:03.711349    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-78a7d5-29816460-lxchn"
Sep 09 21:00:04.475949 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:04.475891    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-78a7d5-29816460-lxchn"
Sep 09 21:00:04.476163 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:04.475890    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-78a7d5-29816460-lxchn" event={"ID":"237bade6-8f58-45a4-8aa6-bdc9a3d03ed4","Type":"ContainerDied","Data":"5c4ad760007a90b84c35cbf7eb7490ce7a05915a5cdc251b6b4307a7a7a0b453"}
Sep 09 21:00:04.476163 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:04.476071    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c4ad760007a90b84c35cbf7eb7490ce7a05915a5cdc251b6b4307a7a7a0b453"
Sep 09 21:00:04.477861 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:04.477703    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-933e9d-29816460-kjcxg" event={"ID":"c1a66023-786a-4984-914e-a15cb5fc394c","Type":"ContainerDied","Data":"ef00474a857ea37092d936206a33644937d960f21ed2b3c62a9366e539a23f2f"}
Sep 09 21:00:04.477861 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:04.477746    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef00474a857ea37092d936206a33644937d960f21ed2b3c62a9366e539a23f2f"
Sep 09 21:00:04.478054 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:04.478002    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-933e9d-29816460-kjcxg"
Sep 09 21:00:04.479198 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:04.479175    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cb82f2-29816460-m4knn"
Sep 09 21:00:04.479198 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:04.479184    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cb82f2-29816460-m4knn" event={"ID":"105b2c9d-1d2c-49a5-b97e-7bb11473a158","Type":"ContainerDied","Data":"90c6ef2166c45429576262b11f56cbe33bfbeaadac0ddd2caa141eb92c643f00"}
Sep 09 21:00:04.479366 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:04.479223    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90c6ef2166c45429576262b11f56cbe33bfbeaadac0ddd2caa141eb92c643f00"
Sep 09 21:00:04.480636 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:04.480613    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fa05f-29816460-b6p2v" event={"ID":"a8539a8e-cacb-4201-9397-300fbb23edec","Type":"ContainerDied","Data":"8209e2495fad091df2f3d48e9d3d5e0b2e21d06d89356d5255d82ae04861fa49"}
Sep 09 21:00:04.480737 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:04.480648    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8209e2495fad091df2f3d48e9d3d5e0b2e21d06d89356d5255d82ae04861fa49"
Sep 09 21:00:04.480737 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:04.480689    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fa05f-29816460-b6p2v"
Sep 09 21:00:04.661102 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:04.661064    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-78a7d5-29816460-lxchn"]
Sep 09 21:00:04.663204 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:04.663183    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-78a7d5-29816460-lxchn"]
Sep 09 21:00:04.698535 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:04.698503    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2"]
Sep 09 21:00:04.698781 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:04.698754    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2" podUID="b7d8f338-b592-4c38-93d5-d41bbefe9c63" containerName="payload-processing" containerID="cri-o://8a9504e02d239c3e7af826456f095c236bb38b52fa43d79f794d30038113617f" gracePeriod=30
Sep 09 21:00:04.894892 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:04.894863    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="237bade6-8f58-45a4-8aa6-bdc9a3d03ed4" path="/var/lib/kubelet/pods/237bade6-8f58-45a4-8aa6-bdc9a3d03ed4/volumes"
Sep 09 21:00:04.909974 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:04.909948    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-78a7d5-lg2hp"]
Sep 09 21:00:04.916471 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:04.916436    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-78a7d5-lg2hp"]
Sep 09 21:00:04.944798 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:04.944780    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2"
Sep 09 21:00:05.065441 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:05.065357    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7d8f338-b592-4c38-93d5-d41bbefe9c63-plugins-config-volume\") pod \"b7d8f338-b592-4c38-93d5-d41bbefe9c63\" (UID: \"b7d8f338-b592-4c38-93d5-d41bbefe9c63\") "
Sep 09 21:00:05.065441 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:05.065397    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndcrg\" (UniqueName: \"kubernetes.io/projected/b7d8f338-b592-4c38-93d5-d41bbefe9c63-kube-api-access-ndcrg\") pod \"b7d8f338-b592-4c38-93d5-d41bbefe9c63\" (UID: \"b7d8f338-b592-4c38-93d5-d41bbefe9c63\") "
Sep 09 21:00:05.065765 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:05.065735    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d8f338-b592-4c38-93d5-d41bbefe9c63-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7d8f338-b592-4c38-93d5-d41bbefe9c63" (UID: "b7d8f338-b592-4c38-93d5-d41bbefe9c63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:00:05.066211 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:05.066179    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7d8f338-b592-4c38-93d5-d41bbefe9c63-plugins-config-volume\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:00:05.067637 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:05.067605    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d8f338-b592-4c38-93d5-d41bbefe9c63-kube-api-access-ndcrg" (OuterVolumeSpecName: "kube-api-access-ndcrg") pod "b7d8f338-b592-4c38-93d5-d41bbefe9c63" (UID: "b7d8f338-b592-4c38-93d5-d41bbefe9c63"). InnerVolumeSpecName "kube-api-access-ndcrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:00:05.167680 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:05.167651    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndcrg\" (UniqueName: \"kubernetes.io/projected/b7d8f338-b592-4c38-93d5-d41bbefe9c63-kube-api-access-ndcrg\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:00:05.487180 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:05.487151    2556 generic.go:356] "Generic (PLEG): container finished" podID="b7d8f338-b592-4c38-93d5-d41bbefe9c63" containerID="8a9504e02d239c3e7af826456f095c236bb38b52fa43d79f794d30038113617f" exitCode=0
Sep 09 21:00:05.487337 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:05.487197    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2"
Sep 09 21:00:05.487337 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:05.487236    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2" event={"ID":"b7d8f338-b592-4c38-93d5-d41bbefe9c63","Type":"ContainerDied","Data":"8a9504e02d239c3e7af826456f095c236bb38b52fa43d79f794d30038113617f"}
Sep 09 21:00:05.487337 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:05.487287    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2" event={"ID":"b7d8f338-b592-4c38-93d5-d41bbefe9c63","Type":"ContainerDied","Data":"1135ba3880314decf932f2c44948df4883b6ffcadc62ed9bc19e600993694aba"}
Sep 09 21:00:05.487337 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:05.487309    2556 scope.go:117] "RemoveContainer" containerID="8a9504e02d239c3e7af826456f095c236bb38b52fa43d79f794d30038113617f"
Sep 09 21:00:05.497047 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:05.497031    2556 scope.go:117] "RemoveContainer" containerID="8a9504e02d239c3e7af826456f095c236bb38b52fa43d79f794d30038113617f"
Sep 09 21:00:05.497276 ip-10-0-135-117 kubenswrapper[2556]: E0909 21:00:05.497259    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9504e02d239c3e7af826456f095c236bb38b52fa43d79f794d30038113617f\": container with ID starting with 8a9504e02d239c3e7af826456f095c236bb38b52fa43d79f794d30038113617f not found: ID does not exist" containerID="8a9504e02d239c3e7af826456f095c236bb38b52fa43d79f794d30038113617f"
Sep 09 21:00:05.497339 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:05.497283    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9504e02d239c3e7af826456f095c236bb38b52fa43d79f794d30038113617f"} err="failed to get container status \"8a9504e02d239c3e7af826456f095c236bb38b52fa43d79f794d30038113617f\": rpc error: code = NotFound desc = could not find container \"8a9504e02d239c3e7af826456f095c236bb38b52fa43d79f794d30038113617f\": container with ID starting with 8a9504e02d239c3e7af826456f095c236bb38b52fa43d79f794d30038113617f not found: ID does not exist"
Sep 09 21:00:05.511525 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:05.511503    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2"]
Sep 09 21:00:05.516211 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:05.516191    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-78a7d5-7984758889-hlnp2"]
Sep 09 21:00:06.893359 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:06.893316    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7d8f338-b592-4c38-93d5-d41bbefe9c63" path="/var/lib/kubelet/pods/b7d8f338-b592-4c38-93d5-d41bbefe9c63/volumes"
Sep 09 21:00:06.893834 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:06.893813    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea738aa3-9847-4681-9e2c-5b64f39a58b6" path="/var/lib/kubelet/pods/ea738aa3-9847-4681-9e2c-5b64f39a58b6/volumes"
Sep 09 21:00:11.805492 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:11.805452    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fa05f-wvh9b"]
Sep 09 21:00:11.805934 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:11.805915    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7d8f338-b592-4c38-93d5-d41bbefe9c63" containerName="payload-processing"
Sep 09 21:00:11.805982 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:11.805934    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7d8f338-b592-4c38-93d5-d41bbefe9c63" containerName="payload-processing"
Sep 09 21:00:11.805982 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:11.805955    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="237bade6-8f58-45a4-8aa6-bdc9a3d03ed4" containerName="cleanup"
Sep 09 21:00:11.805982 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:11.805961    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="237bade6-8f58-45a4-8aa6-bdc9a3d03ed4" containerName="cleanup"
Sep 09 21:00:11.805982 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:11.805969    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a66023-786a-4984-914e-a15cb5fc394c" containerName="cleanup"
Sep 09 21:00:11.805982 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:11.805976    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a66023-786a-4984-914e-a15cb5fc394c" containerName="cleanup"
Sep 09 21:00:11.805982 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:11.805983    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8539a8e-cacb-4201-9397-300fbb23edec" containerName="cleanup"
Sep 09 21:00:11.806175 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:11.805988    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8539a8e-cacb-4201-9397-300fbb23edec" containerName="cleanup"
Sep 09 21:00:11.806175 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:11.805995    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="105b2c9d-1d2c-49a5-b97e-7bb11473a158" containerName="cleanup"
Sep 09 21:00:11.806175 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:11.806000    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="105b2c9d-1d2c-49a5-b97e-7bb11473a158" containerName="cleanup"
Sep 09 21:00:11.806175 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:11.806092    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8539a8e-cacb-4201-9397-300fbb23edec" containerName="cleanup"
Sep 09 21:00:11.806175 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:11.806103    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="237bade6-8f58-45a4-8aa6-bdc9a3d03ed4" containerName="cleanup"
Sep 09 21:00:11.806175 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:11.806110    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1a66023-786a-4984-914e-a15cb5fc394c" containerName="cleanup"
Sep 09 21:00:11.806175 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:11.806117    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7d8f338-b592-4c38-93d5-d41bbefe9c63" containerName="payload-processing"
Sep 09 21:00:11.806175 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:11.806125    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="105b2c9d-1d2c-49a5-b97e-7bb11473a158" containerName="cleanup"
Sep 09 21:00:11.809425 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:11.809410    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fa05f-wvh9b"
Sep 09 21:00:11.812226 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:11.812195    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsks9\""
Sep 09 21:00:11.812226 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:11.812208    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 21:00:11.817235 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:11.817211    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fa05f-wvh9b"]
Sep 09 21:00:11.828992 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:11.828969    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cba8f71-27c1-4289-8912-62c48b9d13ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3fa05f-wvh9b\" (UID: \"9cba8f71-27c1-4289-8912-62c48b9d13ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fa05f-wvh9b"
Sep 09 21:00:11.930287 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:11.930253    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cba8f71-27c1-4289-8912-62c48b9d13ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3fa05f-wvh9b\" (UID: \"9cba8f71-27c1-4289-8912-62c48b9d13ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fa05f-wvh9b"
Sep 09 21:00:11.930798 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:11.930780    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cba8f71-27c1-4289-8912-62c48b9d13ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3fa05f-wvh9b\" (UID: \"9cba8f71-27c1-4289-8912-62c48b9d13ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fa05f-wvh9b"
Sep 09 21:00:12.121031 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:12.120953    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fa05f-wvh9b"
Sep 09 21:00:12.394547 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:12.394520    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fa05f-wvh9b"]
Sep 09 21:00:12.396467 ip-10-0-135-117 kubenswrapper[2556]: W0909 21:00:12.396430    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cba8f71_27c1_4289_8912_62c48b9d13ef.slice/crio-85b81e49a50de4d014b61f0639c95d4946f01ab2587ace7684c8b49bdbb77182 WatchSource:0}: Error finding container 85b81e49a50de4d014b61f0639c95d4946f01ab2587ace7684c8b49bdbb77182: Status 404 returned error can't find the container with id 85b81e49a50de4d014b61f0639c95d4946f01ab2587ace7684c8b49bdbb77182
Sep 09 21:00:12.533443 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:12.533416    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fa05f-wvh9b" event={"ID":"9cba8f71-27c1-4289-8912-62c48b9d13ef","Type":"ContainerStarted","Data":"85b81e49a50de4d014b61f0639c95d4946f01ab2587ace7684c8b49bdbb77182"}
Sep 09 21:00:13.548979 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:13.548943    2556 generic.go:356] "Generic (PLEG): container finished" podID="9cba8f71-27c1-4289-8912-62c48b9d13ef" containerID="00c390eaf016954a21de85e340f2928c30ffd1f2bbb5ca79c8c06346fc922699" exitCode=0
Sep 09 21:00:13.549336 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:13.549044    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fa05f-wvh9b" event={"ID":"9cba8f71-27c1-4289-8912-62c48b9d13ef","Type":"ContainerDied","Data":"00c390eaf016954a21de85e340f2928c30ffd1f2bbb5ca79c8c06346fc922699"}
Sep 09 21:00:14.687525 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:14.687497    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fa05f-wvh9b"
Sep 09 21:00:14.759229 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:14.759205    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cba8f71-27c1-4289-8912-62c48b9d13ef-maas-api-service-ca\") pod \"9cba8f71-27c1-4289-8912-62c48b9d13ef\" (UID: \"9cba8f71-27c1-4289-8912-62c48b9d13ef\") "
Sep 09 21:00:14.759523 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:14.759500    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cba8f71-27c1-4289-8912-62c48b9d13ef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9cba8f71-27c1-4289-8912-62c48b9d13ef" (UID: "9cba8f71-27c1-4289-8912-62c48b9d13ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:00:14.860677 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:14.860604    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cba8f71-27c1-4289-8912-62c48b9d13ef-maas-api-service-ca\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:00:15.562337 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:15.562309    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fa05f-wvh9b"
Sep 09 21:00:15.562498 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:15.562307    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fa05f-wvh9b" event={"ID":"9cba8f71-27c1-4289-8912-62c48b9d13ef","Type":"ContainerDied","Data":"85b81e49a50de4d014b61f0639c95d4946f01ab2587ace7684c8b49bdbb77182"}
Sep 09 21:00:15.562498 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:15.562428    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85b81e49a50de4d014b61f0639c95d4946f01ab2587ace7684c8b49bdbb77182"
Sep 09 21:00:22.314368 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:22.314333    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:00:22.395347 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:22.395298    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fa05f-29816460-b6p2v"]
Sep 09 21:00:22.406560 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:22.406531    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fa05f-29816460-b6p2v"]
Sep 09 21:00:22.629187 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:22.629115    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fa05f-wvh9b"]
Sep 09 21:00:22.632131 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:22.632106    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fa05f-wvh9b"]
Sep 09 21:00:22.893245 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:22.893167    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cba8f71-27c1-4289-8912-62c48b9d13ef" path="/var/lib/kubelet/pods/9cba8f71-27c1-4289-8912-62c48b9d13ef/volumes"
Sep 09 21:00:22.893453 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:22.893440    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8539a8e-cacb-4201-9397-300fbb23edec" path="/var/lib/kubelet/pods/a8539a8e-cacb-4201-9397-300fbb23edec/volumes"
Sep 09 21:00:29.084649 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:29.084615    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"]
Sep 09 21:00:29.085077 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:29.084812    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn" podUID="5fc928d9-3ea3-46bd-a1d1-def8adefab3d" containerName="istio-proxy" containerID="cri-o://4711f7e472deca428d5c30c98d91604af908c2a2b76286c7f85946ebcc648be6" gracePeriod=30
Sep 09 21:00:31.850492 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:31.850458    2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 21:00:31.850938 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:31.850550    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn" podUID="5fc928d9-3ea3-46bd-a1d1-def8adefab3d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:00:34.339221 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.339196    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 21:00:34.440318 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.440245    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-credential-socket\") pod \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") "
Sep 09 21:00:34.440318 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.440279    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-workload-socket\") pod \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") "
Sep 09 21:00:34.440318 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.440313    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-token\") pod \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") "
Sep 09 21:00:34.440579 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.440361    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-data\") pod \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") "
Sep 09 21:00:34.440579 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.440417    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ls4n8\" (UniqueName: \"kubernetes.io/projected/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-kube-api-access-ls4n8\") pod \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") "
Sep 09 21:00:34.440579 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.440476    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-podinfo\") pod \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") "
Sep 09 21:00:34.440579 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.440511    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-workload-certs\") pod \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") "
Sep 09 21:00:34.440579 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.440530    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5fc928d9-3ea3-46bd-a1d1-def8adefab3d" (UID: "5fc928d9-3ea3-46bd-a1d1-def8adefab3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:34.440898 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.440578    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5fc928d9-3ea3-46bd-a1d1-def8adefab3d" (UID: "5fc928d9-3ea3-46bd-a1d1-def8adefab3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:34.440898 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.440600    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istiod-ca-cert\") pod \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") "
Sep 09 21:00:34.440898 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.440647    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-envoy\") pod \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\" (UID: \"5fc928d9-3ea3-46bd-a1d1-def8adefab3d\") "
Sep 09 21:00:34.440898 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.440726    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-data" (OuterVolumeSpecName: "istio-data") pod "5fc928d9-3ea3-46bd-a1d1-def8adefab3d" (UID: "5fc928d9-3ea3-46bd-a1d1-def8adefab3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:34.440898 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.440818    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5fc928d9-3ea3-46bd-a1d1-def8adefab3d" (UID: "5fc928d9-3ea3-46bd-a1d1-def8adefab3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:34.441192 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.440991    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5fc928d9-3ea3-46bd-a1d1-def8adefab3d" (UID: "5fc928d9-3ea3-46bd-a1d1-def8adefab3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:00:34.441192 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.441158    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istiod-ca-cert\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:00:34.441192 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.441176    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-credential-socket\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:00:34.441192 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.441186    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-workload-socket\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:00:34.441192 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.441194    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-data\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:00:34.441458 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.441203    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-workload-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:00:34.442735 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.442711    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5fc928d9-3ea3-46bd-a1d1-def8adefab3d" (UID: "5fc928d9-3ea3-46bd-a1d1-def8adefab3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:34.442837 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.442752    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5fc928d9-3ea3-46bd-a1d1-def8adefab3d" (UID: "5fc928d9-3ea3-46bd-a1d1-def8adefab3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:00:34.443055 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.443032    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-token" (OuterVolumeSpecName: "istio-token") pod "5fc928d9-3ea3-46bd-a1d1-def8adefab3d" (UID: "5fc928d9-3ea3-46bd-a1d1-def8adefab3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:00:34.443171 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.443048    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-kube-api-access-ls4n8" (OuterVolumeSpecName: "kube-api-access-ls4n8") pod "5fc928d9-3ea3-46bd-a1d1-def8adefab3d" (UID: "5fc928d9-3ea3-46bd-a1d1-def8adefab3d"). InnerVolumeSpecName "kube-api-access-ls4n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:00:34.542349 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.542324    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-envoy\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:00:34.542349 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.542345    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-token\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:00:34.542497 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.542355    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls4n8\" (UniqueName: \"kubernetes.io/projected/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-kube-api-access-ls4n8\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:00:34.542497 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.542363    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fc928d9-3ea3-46bd-a1d1-def8adefab3d-istio-podinfo\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:00:34.676927 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.676889    2556 generic.go:356] "Generic (PLEG): container finished" podID="5fc928d9-3ea3-46bd-a1d1-def8adefab3d" containerID="4711f7e472deca428d5c30c98d91604af908c2a2b76286c7f85946ebcc648be6" exitCode=0
Sep 09 21:00:34.677030 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.676943    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn" event={"ID":"5fc928d9-3ea3-46bd-a1d1-def8adefab3d","Type":"ContainerDied","Data":"4711f7e472deca428d5c30c98d91604af908c2a2b76286c7f85946ebcc648be6"}
Sep 09 21:00:34.677030 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.676958    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"
Sep 09 21:00:34.677030 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.677007    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn" event={"ID":"5fc928d9-3ea3-46bd-a1d1-def8adefab3d","Type":"ContainerDied","Data":"ae672042a4874c4487f81121b5dd784ed710fbb5cae2b849cce03d173b205176"}
Sep 09 21:00:34.677136 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.677038    2556 scope.go:117] "RemoveContainer" containerID="4711f7e472deca428d5c30c98d91604af908c2a2b76286c7f85946ebcc648be6"
Sep 09 21:00:34.686875 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.686857    2556 scope.go:117] "RemoveContainer" containerID="3b56a080d72573dacf3980751b790a5dde8a78addc97baace6966c386023e576"
Sep 09 21:00:34.697707 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.697673    2556 scope.go:117] "RemoveContainer" containerID="4711f7e472deca428d5c30c98d91604af908c2a2b76286c7f85946ebcc648be6"
Sep 09 21:00:34.697979 ip-10-0-135-117 kubenswrapper[2556]: E0909 21:00:34.697961    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4711f7e472deca428d5c30c98d91604af908c2a2b76286c7f85946ebcc648be6\": container with ID starting with 4711f7e472deca428d5c30c98d91604af908c2a2b76286c7f85946ebcc648be6 not found: ID does not exist" containerID="4711f7e472deca428d5c30c98d91604af908c2a2b76286c7f85946ebcc648be6"
Sep 09 21:00:34.698052 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.697987    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4711f7e472deca428d5c30c98d91604af908c2a2b76286c7f85946ebcc648be6"} err="failed to get container status \"4711f7e472deca428d5c30c98d91604af908c2a2b76286c7f85946ebcc648be6\": rpc error: code = NotFound desc = could not find container \"4711f7e472deca428d5c30c98d91604af908c2a2b76286c7f85946ebcc648be6\": container with ID starting with 4711f7e472deca428d5c30c98d91604af908c2a2b76286c7f85946ebcc648be6 not found: ID does not exist"
Sep 09 21:00:34.698052 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.698004    2556 scope.go:117] "RemoveContainer" containerID="3b56a080d72573dacf3980751b790a5dde8a78addc97baace6966c386023e576"
Sep 09 21:00:34.698288 ip-10-0-135-117 kubenswrapper[2556]: E0909 21:00:34.698271    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b56a080d72573dacf3980751b790a5dde8a78addc97baace6966c386023e576\": container with ID starting with 3b56a080d72573dacf3980751b790a5dde8a78addc97baace6966c386023e576 not found: ID does not exist" containerID="3b56a080d72573dacf3980751b790a5dde8a78addc97baace6966c386023e576"
Sep 09 21:00:34.698336 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.698293    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b56a080d72573dacf3980751b790a5dde8a78addc97baace6966c386023e576"} err="failed to get container status \"3b56a080d72573dacf3980751b790a5dde8a78addc97baace6966c386023e576\": rpc error: code = NotFound desc = could not find container \"3b56a080d72573dacf3980751b790a5dde8a78addc97baace6966c386023e576\": container with ID starting with 3b56a080d72573dacf3980751b790a5dde8a78addc97baace6966c386023e576 not found: ID does not exist"
Sep 09 21:00:34.703389 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.703368    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"]
Sep 09 21:00:34.710334 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.710313    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3fa05f-openshift-default-574c4fc8cd-dzsnn"]
Sep 09 21:00:34.893388 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.893362    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fc928d9-3ea3-46bd-a1d1-def8adefab3d" path="/var/lib/kubelet/pods/5fc928d9-3ea3-46bd-a1d1-def8adefab3d/volumes"
Sep 09 21:00:34.968406 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:34.968352    2556 scope.go:117] "RemoveContainer" containerID="7d650253985f12ab684dbab682cc5ef09f45adfc29e3f2e6afbd55eb539b4212"
Sep 09 21:00:52.594278 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:00:52.594245    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:01:19.006528 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:19.006499    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:01:32.343538 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:32.343503    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"]
Sep 09 21:01:47.363488 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:47.363431    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj" podUID="40ee8777-c711-4e59-ab09-91eb3d62d56d" containerName="main" containerID="cri-o://2f02d9a5fe275bb2f73c7906e02a049a7b736bdde02c5a152267306380b86353" gracePeriod=45
Sep 09 21:01:47.613673 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:47.613612    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 21:01:47.780152 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:47.780114    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ee8777-c711-4e59-ab09-91eb3d62d56d-tls-certs\") pod \"40ee8777-c711-4e59-ab09-91eb3d62d56d\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") "
Sep 09 21:01:47.780152 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:47.780164    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-tmp-dir\") pod \"40ee8777-c711-4e59-ab09-91eb3d62d56d\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") "
Sep 09 21:01:47.780361 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:47.780207    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-home\") pod \"40ee8777-c711-4e59-ab09-91eb3d62d56d\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") "
Sep 09 21:01:47.780361 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:47.780271    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5pb6\" (UniqueName: \"kubernetes.io/projected/40ee8777-c711-4e59-ab09-91eb3d62d56d-kube-api-access-h5pb6\") pod \"40ee8777-c711-4e59-ab09-91eb3d62d56d\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") "
Sep 09 21:01:47.780361 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:47.780332    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-dshm\") pod \"40ee8777-c711-4e59-ab09-91eb3d62d56d\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") "
Sep 09 21:01:47.780500 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:47.780380    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-model-cache\") pod \"40ee8777-c711-4e59-ab09-91eb3d62d56d\" (UID: \"40ee8777-c711-4e59-ab09-91eb3d62d56d\") "
Sep 09 21:01:47.780500 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:47.780448    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "40ee8777-c711-4e59-ab09-91eb3d62d56d" (UID: "40ee8777-c711-4e59-ab09-91eb3d62d56d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:01:47.780615 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:47.780493    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-home" (OuterVolumeSpecName: "home") pod "40ee8777-c711-4e59-ab09-91eb3d62d56d" (UID: "40ee8777-c711-4e59-ab09-91eb3d62d56d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:01:47.780689 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:47.780656    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-model-cache" (OuterVolumeSpecName: "model-cache") pod "40ee8777-c711-4e59-ab09-91eb3d62d56d" (UID: "40ee8777-c711-4e59-ab09-91eb3d62d56d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:01:47.780871 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:47.780850    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-tmp-dir\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:01:47.780975 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:47.780874    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-home\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:01:47.780975 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:47.780887    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-model-cache\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:01:47.782438 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:47.782411    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40ee8777-c711-4e59-ab09-91eb3d62d56d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40ee8777-c711-4e59-ab09-91eb3d62d56d" (UID: "40ee8777-c711-4e59-ab09-91eb3d62d56d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:01:47.782438 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:47.782411    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-dshm" (OuterVolumeSpecName: "dshm") pod "40ee8777-c711-4e59-ab09-91eb3d62d56d" (UID: "40ee8777-c711-4e59-ab09-91eb3d62d56d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:01:47.782568 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:47.782464    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ee8777-c711-4e59-ab09-91eb3d62d56d-kube-api-access-h5pb6" (OuterVolumeSpecName: "kube-api-access-h5pb6") pod "40ee8777-c711-4e59-ab09-91eb3d62d56d" (UID: "40ee8777-c711-4e59-ab09-91eb3d62d56d"). InnerVolumeSpecName "kube-api-access-h5pb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:01:47.882360 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:47.882295    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5pb6\" (UniqueName: \"kubernetes.io/projected/40ee8777-c711-4e59-ab09-91eb3d62d56d-kube-api-access-h5pb6\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:01:47.882360 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:47.882320    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ee8777-c711-4e59-ab09-91eb3d62d56d-dshm\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:01:47.882360 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:47.882335    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ee8777-c711-4e59-ab09-91eb3d62d56d-tls-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:01:48.094511 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:48.094477    2556 generic.go:356] "Generic (PLEG): container finished" podID="40ee8777-c711-4e59-ab09-91eb3d62d56d" containerID="2f02d9a5fe275bb2f73c7906e02a049a7b736bdde02c5a152267306380b86353" exitCode=0
Sep 09 21:01:48.094625 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:48.094521    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"
Sep 09 21:01:48.094625 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:48.094528    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj" event={"ID":"40ee8777-c711-4e59-ab09-91eb3d62d56d","Type":"ContainerDied","Data":"2f02d9a5fe275bb2f73c7906e02a049a7b736bdde02c5a152267306380b86353"}
Sep 09 21:01:48.094625 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:48.094566    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj" event={"ID":"40ee8777-c711-4e59-ab09-91eb3d62d56d","Type":"ContainerDied","Data":"b7d28df7b581f361be1d7d5dff4008f2457811095ce436a13de9262655565bcb"}
Sep 09 21:01:48.094625 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:48.094588    2556 scope.go:117] "RemoveContainer" containerID="2f02d9a5fe275bb2f73c7906e02a049a7b736bdde02c5a152267306380b86353"
Sep 09 21:01:48.104287 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:48.104270    2556 scope.go:117] "RemoveContainer" containerID="2f02d9a5fe275bb2f73c7906e02a049a7b736bdde02c5a152267306380b86353"
Sep 09 21:01:48.104575 ip-10-0-135-117 kubenswrapper[2556]: E0909 21:01:48.104543    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f02d9a5fe275bb2f73c7906e02a049a7b736bdde02c5a152267306380b86353\": container with ID starting with 2f02d9a5fe275bb2f73c7906e02a049a7b736bdde02c5a152267306380b86353 not found: ID does not exist" containerID="2f02d9a5fe275bb2f73c7906e02a049a7b736bdde02c5a152267306380b86353"
Sep 09 21:01:48.104629 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:48.104572    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f02d9a5fe275bb2f73c7906e02a049a7b736bdde02c5a152267306380b86353"} err="failed to get container status \"2f02d9a5fe275bb2f73c7906e02a049a7b736bdde02c5a152267306380b86353\": rpc error: code = NotFound desc = could not find container \"2f02d9a5fe275bb2f73c7906e02a049a7b736bdde02c5a152267306380b86353\": container with ID starting with 2f02d9a5fe275bb2f73c7906e02a049a7b736bdde02c5a152267306380b86353 not found: ID does not exist"
Sep 09 21:01:48.118095 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:48.118072    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"]
Sep 09 21:01:48.122644 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:48.122619    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/e2e-conflict-a-dff9f838-kserve-785786567f-7wtwj"]
Sep 09 21:01:48.896925 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:48.896882    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40ee8777-c711-4e59-ab09-91eb3d62d56d" path="/var/lib/kubelet/pods/40ee8777-c711-4e59-ab09-91eb3d62d56d/volumes"
Sep 09 21:01:53.600521 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:01:53.600482    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:02:17.793439 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:17.793402    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:02:35.028785 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:35.028753    2556 scope.go:117] "RemoveContainer" containerID="75d4ced149c8c0c564aff588ba4d407c7048a175b4b7544b280a7a32ca9698ac"
Sep 09 21:02:39.782305 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:39.782269    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"]
Sep 09 21:02:40.100972 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:40.100857    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:02:43.305452 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.305422    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"]
Sep 09 21:02:43.305849 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.305834    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fc928d9-3ea3-46bd-a1d1-def8adefab3d" containerName="istio-proxy"
Sep 09 21:02:43.305893 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.305851    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fc928d9-3ea3-46bd-a1d1-def8adefab3d" containerName="istio-proxy"
Sep 09 21:02:43.305893 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.305869    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40ee8777-c711-4e59-ab09-91eb3d62d56d" containerName="main"
Sep 09 21:02:43.305893 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.305875    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="40ee8777-c711-4e59-ab09-91eb3d62d56d" containerName="main"
Sep 09 21:02:43.305893 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.305893    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cba8f71-27c1-4289-8912-62c48b9d13ef" containerName="revoke-keys"
Sep 09 21:02:43.306089 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.305899    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cba8f71-27c1-4289-8912-62c48b9d13ef" containerName="revoke-keys"
Sep 09 21:02:43.306089 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.305980    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cba8f71-27c1-4289-8912-62c48b9d13ef" containerName="revoke-keys"
Sep 09 21:02:43.306089 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.305990    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fc928d9-3ea3-46bd-a1d1-def8adefab3d" containerName="istio-proxy"
Sep 09 21:02:43.306089 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.306003    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="40ee8777-c711-4e59-ab09-91eb3d62d56d" containerName="main"
Sep 09 21:02:43.308178 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.308158    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:43.312933 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.312880    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-cb82f2\"/\"default-dockercfg-bjtcp\""
Sep 09 21:02:43.313069 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.312948    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-cb82f2\"/\"sub-test-model-cb82f2-kserve-self-signed-certs\""
Sep 09 21:02:43.313069 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.312951    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-cb82f2\"/\"kube-root-ca.crt\""
Sep 09 21:02:43.313167 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.313112    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-cb82f2\"/\"openshift-service-ca.crt\""
Sep 09 21:02:43.320637 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.320617    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"]
Sep 09 21:02:43.361695 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.361675    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-dshm\") pod \"sub-test-model-cb82f2-kserve-7688b596cc-qzpvx\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") " pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:43.361811 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.361708    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86tpp\" (UniqueName: \"kubernetes.io/projected/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-kube-api-access-86tpp\") pod \"sub-test-model-cb82f2-kserve-7688b596cc-qzpvx\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") " pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:43.361811 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.361738    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-model-cache\") pod \"sub-test-model-cb82f2-kserve-7688b596cc-qzpvx\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") " pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:43.361811 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.361800    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-tmp-dir\") pod \"sub-test-model-cb82f2-kserve-7688b596cc-qzpvx\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") " pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:43.361980 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.361856    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-home\") pod \"sub-test-model-cb82f2-kserve-7688b596cc-qzpvx\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") " pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:43.361980 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.361893    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-tls-certs\") pod \"sub-test-model-cb82f2-kserve-7688b596cc-qzpvx\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") " pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:43.462699 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.462681    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-dshm\") pod \"sub-test-model-cb82f2-kserve-7688b596cc-qzpvx\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") " pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:43.462801 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.462727    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86tpp\" (UniqueName: \"kubernetes.io/projected/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-kube-api-access-86tpp\") pod \"sub-test-model-cb82f2-kserve-7688b596cc-qzpvx\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") " pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:43.462801 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.462755    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-model-cache\") pod \"sub-test-model-cb82f2-kserve-7688b596cc-qzpvx\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") " pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:43.462801 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.462777    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-tmp-dir\") pod \"sub-test-model-cb82f2-kserve-7688b596cc-qzpvx\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") " pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:43.462987 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.462802    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-home\") pod \"sub-test-model-cb82f2-kserve-7688b596cc-qzpvx\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") " pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:43.462987 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.462827    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-tls-certs\") pod \"sub-test-model-cb82f2-kserve-7688b596cc-qzpvx\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") " pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:43.463198 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.463180    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-tmp-dir\") pod \"sub-test-model-cb82f2-kserve-7688b596cc-qzpvx\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") " pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:43.463283 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.463207    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-home\") pod \"sub-test-model-cb82f2-kserve-7688b596cc-qzpvx\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") " pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:43.463283 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.463246    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-model-cache\") pod \"sub-test-model-cb82f2-kserve-7688b596cc-qzpvx\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") " pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:43.465014 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.464994    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-dshm\") pod \"sub-test-model-cb82f2-kserve-7688b596cc-qzpvx\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") " pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:43.465187 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.465170    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-tls-certs\") pod \"sub-test-model-cb82f2-kserve-7688b596cc-qzpvx\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") " pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:43.470962 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.470943    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86tpp\" (UniqueName: \"kubernetes.io/projected/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-kube-api-access-86tpp\") pod \"sub-test-model-cb82f2-kserve-7688b596cc-qzpvx\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") " pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:43.617219 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.617140    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:43.889306 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:43.889282    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"]
Sep 09 21:02:43.891492 ip-10-0-135-117 kubenswrapper[2556]: W0909 21:02:43.891466    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod114f5b2b_f308_4c81_a3ca_f8ccb87c08cf.slice/crio-eff43cc48e2624e2670a94e05a4943aa5d155940f9021ce73769adc182b6d6cc WatchSource:0}: Error finding container eff43cc48e2624e2670a94e05a4943aa5d155940f9021ce73769adc182b6d6cc: Status 404 returned error can't find the container with id eff43cc48e2624e2670a94e05a4943aa5d155940f9021ce73769adc182b6d6cc
Sep 09 21:02:44.417754 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:44.417717    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx" event={"ID":"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf","Type":"ContainerStarted","Data":"a4a6b2ef16c33e828ec4c50db5f97c031c0cf955217369abb8879c89ec2e8d0f"}
Sep 09 21:02:44.418141 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:44.417767    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx" event={"ID":"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf","Type":"ContainerStarted","Data":"eff43cc48e2624e2670a94e05a4943aa5d155940f9021ce73769adc182b6d6cc"}
Sep 09 21:02:44.436081 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:44.436036    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx" podStartSLOduration=1.4360199599999999 podStartE2EDuration="1.43601996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:02:44.434632166 +0000 UTC m=+1810.114592477" watchObservedRunningTime="2026-09-09 21:02:44.43601996 +0000 UTC m=+1810.115980269"
Sep 09 21:02:53.618302 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:53.618267    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:53.618834 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:53.618318    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:53.630600 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:53.630565    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:54.490458 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:54.490429    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:02:54.801556 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:54.801468    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj" podUID="06060449-719b-4544-a976-30268aede890" containerName="main" containerID="cri-o://815a84a69fa8a3bd088c2b33d3e369dabbced367abd5f511b3659d8f2552a070" gracePeriod=45
Sep 09 21:02:55.050261 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.050239    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 21:02:55.166529 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.166459    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-model-cache\") pod \"06060449-719b-4544-a976-30268aede890\" (UID: \"06060449-719b-4544-a976-30268aede890\") "
Sep 09 21:02:55.166529 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.166492    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06060449-719b-4544-a976-30268aede890-tls-certs\") pod \"06060449-719b-4544-a976-30268aede890\" (UID: \"06060449-719b-4544-a976-30268aede890\") "
Sep 09 21:02:55.166697 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.166583    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mtfx\" (UniqueName: \"kubernetes.io/projected/06060449-719b-4544-a976-30268aede890-kube-api-access-5mtfx\") pod \"06060449-719b-4544-a976-30268aede890\" (UID: \"06060449-719b-4544-a976-30268aede890\") "
Sep 09 21:02:55.166697 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.166624    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-tmp-dir\") pod \"06060449-719b-4544-a976-30268aede890\" (UID: \"06060449-719b-4544-a976-30268aede890\") "
Sep 09 21:02:55.166697 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.166664    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-home\") pod \"06060449-719b-4544-a976-30268aede890\" (UID: \"06060449-719b-4544-a976-30268aede890\") "
Sep 09 21:02:55.166697 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.166673    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-model-cache" (OuterVolumeSpecName: "model-cache") pod "06060449-719b-4544-a976-30268aede890" (UID: "06060449-719b-4544-a976-30268aede890"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:02:55.166937 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.166683    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-dshm\") pod \"06060449-719b-4544-a976-30268aede890\" (UID: \"06060449-719b-4544-a976-30268aede890\") "
Sep 09 21:02:55.166937 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.166891    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-home" (OuterVolumeSpecName: "home") pod "06060449-719b-4544-a976-30268aede890" (UID: "06060449-719b-4544-a976-30268aede890"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:02:55.167051 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.166940    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06060449-719b-4544-a976-30268aede890" (UID: "06060449-719b-4544-a976-30268aede890"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:02:55.167180 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.167160    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-tmp-dir\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:02:55.167479 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.167185    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-home\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:02:55.167479 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.167199    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-model-cache\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:02:55.168811 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.168790    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-dshm" (OuterVolumeSpecName: "dshm") pod "06060449-719b-4544-a976-30268aede890" (UID: "06060449-719b-4544-a976-30268aede890"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:02:55.168901 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.168834    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06060449-719b-4544-a976-30268aede890-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06060449-719b-4544-a976-30268aede890" (UID: "06060449-719b-4544-a976-30268aede890"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:02:55.168901 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.168876    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06060449-719b-4544-a976-30268aede890-kube-api-access-5mtfx" (OuterVolumeSpecName: "kube-api-access-5mtfx") pod "06060449-719b-4544-a976-30268aede890" (UID: "06060449-719b-4544-a976-30268aede890"). InnerVolumeSpecName "kube-api-access-5mtfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:02:55.268278 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.268255    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06060449-719b-4544-a976-30268aede890-dshm\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:02:55.268278 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.268274    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06060449-719b-4544-a976-30268aede890-tls-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:02:55.268406 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.268285    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mtfx\" (UniqueName: \"kubernetes.io/projected/06060449-719b-4544-a976-30268aede890-kube-api-access-5mtfx\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:02:55.485652 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.485622    2556 generic.go:356] "Generic (PLEG): container finished" podID="06060449-719b-4544-a976-30268aede890" containerID="815a84a69fa8a3bd088c2b33d3e369dabbced367abd5f511b3659d8f2552a070" exitCode=0
Sep 09 21:02:55.485787 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.485667    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"
Sep 09 21:02:55.485787 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.485698    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj" event={"ID":"06060449-719b-4544-a976-30268aede890","Type":"ContainerDied","Data":"815a84a69fa8a3bd088c2b33d3e369dabbced367abd5f511b3659d8f2552a070"}
Sep 09 21:02:55.485787 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.485734    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj" event={"ID":"06060449-719b-4544-a976-30268aede890","Type":"ContainerDied","Data":"977f584ae9b532466d0f781af1c969124b0728f0c1d9ccbb59d6421af4aac1d3"}
Sep 09 21:02:55.485787 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.485751    2556 scope.go:117] "RemoveContainer" containerID="815a84a69fa8a3bd088c2b33d3e369dabbced367abd5f511b3659d8f2552a070"
Sep 09 21:02:55.495452 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.495430    2556 scope.go:117] "RemoveContainer" containerID="815a84a69fa8a3bd088c2b33d3e369dabbced367abd5f511b3659d8f2552a070"
Sep 09 21:02:55.495703 ip-10-0-135-117 kubenswrapper[2556]: E0909 21:02:55.495683    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"815a84a69fa8a3bd088c2b33d3e369dabbced367abd5f511b3659d8f2552a070\": container with ID starting with 815a84a69fa8a3bd088c2b33d3e369dabbced367abd5f511b3659d8f2552a070 not found: ID does not exist" containerID="815a84a69fa8a3bd088c2b33d3e369dabbced367abd5f511b3659d8f2552a070"
Sep 09 21:02:55.495748 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.495711    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815a84a69fa8a3bd088c2b33d3e369dabbced367abd5f511b3659d8f2552a070"} err="failed to get container status \"815a84a69fa8a3bd088c2b33d3e369dabbced367abd5f511b3659d8f2552a070\": rpc error: code = NotFound desc = could not find container \"815a84a69fa8a3bd088c2b33d3e369dabbced367abd5f511b3659d8f2552a070\": container with ID starting with 815a84a69fa8a3bd088c2b33d3e369dabbced367abd5f511b3659d8f2552a070 not found: ID does not exist"
Sep 09 21:02:55.507589 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.507565    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"]
Sep 09 21:02:55.512774 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:55.512756    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a2fb5a/auth-test-model-a2fb5a-kserve-66b76b9c99-2tcnj"]
Sep 09 21:02:56.892864 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:02:56.892833    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06060449-719b-4544-a976-30268aede890" path="/var/lib/kubelet/pods/06060449-719b-4544-a976-30268aede890/volumes"
Sep 09 21:03:03.505990 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:03.505950    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:03:19.742101 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:19.742062    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-933e9d-fkxgs"]
Sep 09 21:03:19.742527 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:19.742474    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fc928d9-3ea3-46bd-a1d1-def8adefab3d" containerName="istio-proxy"
Sep 09 21:03:19.742527 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:19.742485    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fc928d9-3ea3-46bd-a1d1-def8adefab3d" containerName="istio-proxy"
Sep 09 21:03:19.742527 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:19.742496    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06060449-719b-4544-a976-30268aede890" containerName="main"
Sep 09 21:03:19.742527 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:19.742501    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="06060449-719b-4544-a976-30268aede890" containerName="main"
Sep 09 21:03:19.742664 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:19.742585    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="06060449-719b-4544-a976-30268aede890" containerName="main"
Sep 09 21:03:19.742664 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:19.742596    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fc928d9-3ea3-46bd-a1d1-def8adefab3d" containerName="istio-proxy"
Sep 09 21:03:19.745786 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:19.745770    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-933e9d-fkxgs"
Sep 09 21:03:19.748687 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:19.748661    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsks9\""
Sep 09 21:03:19.748846 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:19.748661    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 21:03:19.755721 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:19.755696    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-933e9d-fkxgs"]
Sep 09 21:03:19.881973 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:19.881946    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920ed061-dc28-4adf-b78d-d4aba5fb6b75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-933e9d-fkxgs\" (UID: \"920ed061-dc28-4adf-b78d-d4aba5fb6b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-933e9d-fkxgs"
Sep 09 21:03:19.983078 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:19.983047    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920ed061-dc28-4adf-b78d-d4aba5fb6b75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-933e9d-fkxgs\" (UID: \"920ed061-dc28-4adf-b78d-d4aba5fb6b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-933e9d-fkxgs"
Sep 09 21:03:19.983603 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:19.983582    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920ed061-dc28-4adf-b78d-d4aba5fb6b75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-933e9d-fkxgs\" (UID: \"920ed061-dc28-4adf-b78d-d4aba5fb6b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-933e9d-fkxgs"
Sep 09 21:03:20.058042 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:20.057981    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-933e9d-fkxgs"
Sep 09 21:03:20.328663 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:20.328635    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-933e9d-fkxgs"]
Sep 09 21:03:20.330123 ip-10-0-135-117 kubenswrapper[2556]: W0909 21:03:20.330095    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920ed061_dc28_4adf_b78d_d4aba5fb6b75.slice/crio-30969e6a00b610005a14c03258904439f2d7b28940c745d228cad6ae2bbe53a7 WatchSource:0}: Error finding container 30969e6a00b610005a14c03258904439f2d7b28940c745d228cad6ae2bbe53a7: Status 404 returned error can't find the container with id 30969e6a00b610005a14c03258904439f2d7b28940c745d228cad6ae2bbe53a7
Sep 09 21:03:20.395702 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:20.395673    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:03:20.628006 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:20.627928    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-933e9d-fkxgs" event={"ID":"920ed061-dc28-4adf-b78d-d4aba5fb6b75","Type":"ContainerStarted","Data":"30969e6a00b610005a14c03258904439f2d7b28940c745d228cad6ae2bbe53a7"}
Sep 09 21:03:21.635200 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:21.635164    2556 generic.go:356] "Generic (PLEG): container finished" podID="920ed061-dc28-4adf-b78d-d4aba5fb6b75" containerID="0493b48338e47d0c2dfde51bada13ac485eb45b8ca46883e1af475d5103e07df" exitCode=0
Sep 09 21:03:21.635570 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:21.635214    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-933e9d-fkxgs" event={"ID":"920ed061-dc28-4adf-b78d-d4aba5fb6b75","Type":"ContainerDied","Data":"0493b48338e47d0c2dfde51bada13ac485eb45b8ca46883e1af475d5103e07df"}
Sep 09 21:03:22.776833 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:22.776805    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-933e9d-fkxgs"
Sep 09 21:03:22.910897 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:22.910836    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920ed061-dc28-4adf-b78d-d4aba5fb6b75-maas-api-service-ca\") pod \"920ed061-dc28-4adf-b78d-d4aba5fb6b75\" (UID: \"920ed061-dc28-4adf-b78d-d4aba5fb6b75\") "
Sep 09 21:03:22.911207 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:22.911183    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920ed061-dc28-4adf-b78d-d4aba5fb6b75-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "920ed061-dc28-4adf-b78d-d4aba5fb6b75" (UID: "920ed061-dc28-4adf-b78d-d4aba5fb6b75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:03:23.011892 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:23.011866    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920ed061-dc28-4adf-b78d-d4aba5fb6b75-maas-api-service-ca\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:03:23.649151 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:23.649123    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-933e9d-fkxgs"
Sep 09 21:03:23.649336 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:23.649130    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-933e9d-fkxgs" event={"ID":"920ed061-dc28-4adf-b78d-d4aba5fb6b75","Type":"ContainerDied","Data":"30969e6a00b610005a14c03258904439f2d7b28940c745d228cad6ae2bbe53a7"}
Sep 09 21:03:23.649336 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:23.649241    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30969e6a00b610005a14c03258904439f2d7b28940c745d228cad6ae2bbe53a7"
Sep 09 21:03:24.608374 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:24.608342    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"]
Sep 09 21:03:24.630102 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:24.630073    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"]
Sep 09 21:03:24.648192 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:24.648167    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"]
Sep 09 21:03:35.544761 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:35.544726    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-933e9d-29816460-kjcxg"]
Sep 09 21:03:35.547853 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:35.547827    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-933e9d-29816460-kjcxg"]
Sep 09 21:03:35.781008 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:35.780975    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-933e9d-fkxgs"]
Sep 09 21:03:35.784297 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:35.784274    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-933e9d-fkxgs"]
Sep 09 21:03:36.893503 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:36.893462    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="920ed061-dc28-4adf-b78d-d4aba5fb6b75" path="/var/lib/kubelet/pods/920ed061-dc28-4adf-b78d-d4aba5fb6b75/volumes"
Sep 09 21:03:36.895876 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:36.893763    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1a66023-786a-4984-914e-a15cb5fc394c" path="/var/lib/kubelet/pods/c1a66023-786a-4984-914e-a15cb5fc394c/volumes"
Sep 09 21:03:39.628497 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:39.628437    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77" podUID="ccd2caf3-add9-49bc-a256-ac962f5b9a69" containerName="main" containerID="cri-o://7e111343cec27d8917aaf405e28f0e3611107144aa40492fcebc6d8f84bfc8f6" gracePeriod=45
Sep 09 21:03:39.649829 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:39.649797    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8" podUID="6bc18bdc-1c2e-4414-8a04-30dead944112" containerName="main" containerID="cri-o://2ee2f7841d59af77792edaf2ec2fbdc6aa461167ac72c9296a2b45bb4142584b" gracePeriod=45
Sep 09 21:03:39.667585 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:39.667539    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw" podUID="57b4976c-255a-4d22-884a-2492e4df333d" containerName="main" containerID="cri-o://d6c414d2e7b65807ac0c6d637966a6b073e5d3bf3775c965a54de1a7115c2874" gracePeriod=45
Sep 09 21:03:39.915357 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:39.915334    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 21:03:39.953557 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:39.953534    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 21:03:39.960205 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:39.960183    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 21:03:39.960479 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:39.960463    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-home\") pod \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") "
Sep 09 21:03:39.960559 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:39.960512    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-model-cache\") pod \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") "
Sep 09 21:03:39.960559 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:39.960541    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkzs8\" (UniqueName: \"kubernetes.io/projected/ccd2caf3-add9-49bc-a256-ac962f5b9a69-kube-api-access-xkzs8\") pod \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") "
Sep 09 21:03:39.960677 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:39.960584    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-tmp-dir\") pod \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") "
Sep 09 21:03:39.960677 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:39.960646    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-dshm\") pod \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") "
Sep 09 21:03:39.960786 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:39.960770    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-model-cache" (OuterVolumeSpecName: "model-cache") pod "ccd2caf3-add9-49bc-a256-ac962f5b9a69" (UID: "ccd2caf3-add9-49bc-a256-ac962f5b9a69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:03:39.960844 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:39.960783    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd2caf3-add9-49bc-a256-ac962f5b9a69-tls-certs\") pod \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\" (UID: \"ccd2caf3-add9-49bc-a256-ac962f5b9a69\") "
Sep 09 21:03:39.961008 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:39.960779    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-home" (OuterVolumeSpecName: "home") pod "ccd2caf3-add9-49bc-a256-ac962f5b9a69" (UID: "ccd2caf3-add9-49bc-a256-ac962f5b9a69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:03:39.961008 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:39.960964    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ccd2caf3-add9-49bc-a256-ac962f5b9a69" (UID: "ccd2caf3-add9-49bc-a256-ac962f5b9a69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:03:39.961428 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:39.961260    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-home\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:03:39.961428 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:39.961284    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-model-cache\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:03:39.961428 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:39.961298    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-tmp-dir\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:03:39.963256 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:39.963229    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd2caf3-add9-49bc-a256-ac962f5b9a69-kube-api-access-xkzs8" (OuterVolumeSpecName: "kube-api-access-xkzs8") pod "ccd2caf3-add9-49bc-a256-ac962f5b9a69" (UID: "ccd2caf3-add9-49bc-a256-ac962f5b9a69"). InnerVolumeSpecName "kube-api-access-xkzs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:03:39.963475 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:39.963446    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccd2caf3-add9-49bc-a256-ac962f5b9a69-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ccd2caf3-add9-49bc-a256-ac962f5b9a69" (UID: "ccd2caf3-add9-49bc-a256-ac962f5b9a69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:03:39.963579 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:39.963486    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-dshm" (OuterVolumeSpecName: "dshm") pod "ccd2caf3-add9-49bc-a256-ac962f5b9a69" (UID: "ccd2caf3-add9-49bc-a256-ac962f5b9a69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:03:40.062056 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.062028    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-dshm\") pod \"6bc18bdc-1c2e-4414-8a04-30dead944112\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") "
Sep 09 21:03:40.062212 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.062079    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-model-cache\") pod \"57b4976c-255a-4d22-884a-2492e4df333d\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") "
Sep 09 21:03:40.062212 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.062112    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkbtk\" (UniqueName: \"kubernetes.io/projected/6bc18bdc-1c2e-4414-8a04-30dead944112-kube-api-access-kkbtk\") pod \"6bc18bdc-1c2e-4414-8a04-30dead944112\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") "
Sep 09 21:03:40.062212 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.062153    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-home\") pod \"57b4976c-255a-4d22-884a-2492e4df333d\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") "
Sep 09 21:03:40.062369 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.062252    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc18bdc-1c2e-4414-8a04-30dead944112-tls-certs\") pod \"6bc18bdc-1c2e-4414-8a04-30dead944112\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") "
Sep 09 21:03:40.062369 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.062291    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-home\") pod \"6bc18bdc-1c2e-4414-8a04-30dead944112\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") "
Sep 09 21:03:40.062369 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.062319    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57b4976c-255a-4d22-884a-2492e4df333d-tls-certs\") pod \"57b4976c-255a-4d22-884a-2492e4df333d\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") "
Sep 09 21:03:40.062529 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.062398    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-tmp-dir\") pod \"6bc18bdc-1c2e-4414-8a04-30dead944112\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") "
Sep 09 21:03:40.062529 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.062447    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-dshm\") pod \"57b4976c-255a-4d22-884a-2492e4df333d\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") "
Sep 09 21:03:40.062529 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.062471    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-model-cache\") pod \"6bc18bdc-1c2e-4414-8a04-30dead944112\" (UID: \"6bc18bdc-1c2e-4414-8a04-30dead944112\") "
Sep 09 21:03:40.062529 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.062495    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjb6q\" (UniqueName: \"kubernetes.io/projected/57b4976c-255a-4d22-884a-2492e4df333d-kube-api-access-mjb6q\") pod \"57b4976c-255a-4d22-884a-2492e4df333d\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") "
Sep 09 21:03:40.062736 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.062542    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-tmp-dir\") pod \"57b4976c-255a-4d22-884a-2492e4df333d\" (UID: \"57b4976c-255a-4d22-884a-2492e4df333d\") "
Sep 09 21:03:40.062736 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.062551    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-home" (OuterVolumeSpecName: "home") pod "57b4976c-255a-4d22-884a-2492e4df333d" (UID: "57b4976c-255a-4d22-884a-2492e4df333d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:03:40.062736 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.062690    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-home" (OuterVolumeSpecName: "home") pod "6bc18bdc-1c2e-4414-8a04-30dead944112" (UID: "6bc18bdc-1c2e-4414-8a04-30dead944112"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:03:40.063245 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.062996    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-home\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:03:40.063245 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.063022    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-home\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:03:40.063245 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.063024    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-model-cache" (OuterVolumeSpecName: "model-cache") pod "57b4976c-255a-4d22-884a-2492e4df333d" (UID: "57b4976c-255a-4d22-884a-2492e4df333d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:03:40.063245 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.063038    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkzs8\" (UniqueName: \"kubernetes.io/projected/ccd2caf3-add9-49bc-a256-ac962f5b9a69-kube-api-access-xkzs8\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:03:40.063245 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.063055    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd2caf3-add9-49bc-a256-ac962f5b9a69-dshm\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:03:40.063245 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.063069    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd2caf3-add9-49bc-a256-ac962f5b9a69-tls-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:03:40.063245 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.063191    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57b4976c-255a-4d22-884a-2492e4df333d" (UID: "57b4976c-255a-4d22-884a-2492e4df333d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:03:40.063245 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.063203    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-model-cache" (OuterVolumeSpecName: "model-cache") pod "6bc18bdc-1c2e-4414-8a04-30dead944112" (UID: "6bc18bdc-1c2e-4414-8a04-30dead944112"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:03:40.063741 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.063265    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6bc18bdc-1c2e-4414-8a04-30dead944112" (UID: "6bc18bdc-1c2e-4414-8a04-30dead944112"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:03:40.064694 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.064658    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bc18bdc-1c2e-4414-8a04-30dead944112-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6bc18bdc-1c2e-4414-8a04-30dead944112" (UID: "6bc18bdc-1c2e-4414-8a04-30dead944112"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:03:40.064694 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.064675    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc18bdc-1c2e-4414-8a04-30dead944112-kube-api-access-kkbtk" (OuterVolumeSpecName: "kube-api-access-kkbtk") pod "6bc18bdc-1c2e-4414-8a04-30dead944112" (UID: "6bc18bdc-1c2e-4414-8a04-30dead944112"). InnerVolumeSpecName "kube-api-access-kkbtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:03:40.064834 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.064690    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-dshm" (OuterVolumeSpecName: "dshm") pod "57b4976c-255a-4d22-884a-2492e4df333d" (UID: "57b4976c-255a-4d22-884a-2492e4df333d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:03:40.064894 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.064868    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-dshm" (OuterVolumeSpecName: "dshm") pod "6bc18bdc-1c2e-4414-8a04-30dead944112" (UID: "6bc18bdc-1c2e-4414-8a04-30dead944112"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:03:40.064960 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.064948    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b4976c-255a-4d22-884a-2492e4df333d-kube-api-access-mjb6q" (OuterVolumeSpecName: "kube-api-access-mjb6q") pod "57b4976c-255a-4d22-884a-2492e4df333d" (UID: "57b4976c-255a-4d22-884a-2492e4df333d"). InnerVolumeSpecName "kube-api-access-mjb6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:03:40.065205 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.065185    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57b4976c-255a-4d22-884a-2492e4df333d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57b4976c-255a-4d22-884a-2492e4df333d" (UID: "57b4976c-255a-4d22-884a-2492e4df333d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:03:40.163549 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.163492    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkbtk\" (UniqueName: \"kubernetes.io/projected/6bc18bdc-1c2e-4414-8a04-30dead944112-kube-api-access-kkbtk\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:03:40.163549 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.163515    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc18bdc-1c2e-4414-8a04-30dead944112-tls-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:03:40.163549 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.163525    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57b4976c-255a-4d22-884a-2492e4df333d-tls-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:03:40.163549 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.163532    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-tmp-dir\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:03:40.163549 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.163540    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-dshm\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:03:40.163549 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.163547    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-model-cache\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:03:40.163792 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.163555    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjb6q\" (UniqueName: \"kubernetes.io/projected/57b4976c-255a-4d22-884a-2492e4df333d-kube-api-access-mjb6q\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:03:40.163792 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.163564    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-tmp-dir\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:03:40.163792 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.163573    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc18bdc-1c2e-4414-8a04-30dead944112-dshm\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:03:40.163792 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.163580    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57b4976c-255a-4d22-884a-2492e4df333d-model-cache\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:03:40.748834 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.748793    2556 generic.go:356] "Generic (PLEG): container finished" podID="57b4976c-255a-4d22-884a-2492e4df333d" containerID="d6c414d2e7b65807ac0c6d637966a6b073e5d3bf3775c965a54de1a7115c2874" exitCode=0
Sep 09 21:03:40.749303 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.748857    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"
Sep 09 21:03:40.749303 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.748876    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw" event={"ID":"57b4976c-255a-4d22-884a-2492e4df333d","Type":"ContainerDied","Data":"d6c414d2e7b65807ac0c6d637966a6b073e5d3bf3775c965a54de1a7115c2874"}
Sep 09 21:03:40.749303 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.748943    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw" event={"ID":"57b4976c-255a-4d22-884a-2492e4df333d","Type":"ContainerDied","Data":"873bbc5b52e41cbc00887d3ea4ffd949d7486a8e7ff79282e9c678a1a4d7daeb"}
Sep 09 21:03:40.749303 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.748963    2556 scope.go:117] "RemoveContainer" containerID="d6c414d2e7b65807ac0c6d637966a6b073e5d3bf3775c965a54de1a7115c2874"
Sep 09 21:03:40.750750 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.750722    2556 generic.go:356] "Generic (PLEG): container finished" podID="ccd2caf3-add9-49bc-a256-ac962f5b9a69" containerID="7e111343cec27d8917aaf405e28f0e3611107144aa40492fcebc6d8f84bfc8f6" exitCode=0
Sep 09 21:03:40.750853 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.750781    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"
Sep 09 21:03:40.750853 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.750810    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77" event={"ID":"ccd2caf3-add9-49bc-a256-ac962f5b9a69","Type":"ContainerDied","Data":"7e111343cec27d8917aaf405e28f0e3611107144aa40492fcebc6d8f84bfc8f6"}
Sep 09 21:03:40.750853 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.750844    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77" event={"ID":"ccd2caf3-add9-49bc-a256-ac962f5b9a69","Type":"ContainerDied","Data":"3c8201857136eb9825a1bdbdd764414743b883d8fb0cb2191cfe06f7102f81f1"}
Sep 09 21:03:40.752428 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.752406    2556 generic.go:356] "Generic (PLEG): container finished" podID="6bc18bdc-1c2e-4414-8a04-30dead944112" containerID="2ee2f7841d59af77792edaf2ec2fbdc6aa461167ac72c9296a2b45bb4142584b" exitCode=0
Sep 09 21:03:40.752552 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.752470    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"
Sep 09 21:03:40.752552 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.752498    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8" event={"ID":"6bc18bdc-1c2e-4414-8a04-30dead944112","Type":"ContainerDied","Data":"2ee2f7841d59af77792edaf2ec2fbdc6aa461167ac72c9296a2b45bb4142584b"}
Sep 09 21:03:40.752552 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.752538    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8" event={"ID":"6bc18bdc-1c2e-4414-8a04-30dead944112","Type":"ContainerDied","Data":"06ce7edd694dd793b12af431d15d7b5482f127360d1983a2bb40339501a41527"}
Sep 09 21:03:40.758862 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.758841    2556 scope.go:117] "RemoveContainer" containerID="d6c414d2e7b65807ac0c6d637966a6b073e5d3bf3775c965a54de1a7115c2874"
Sep 09 21:03:40.759347 ip-10-0-135-117 kubenswrapper[2556]: E0909 21:03:40.759326    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6c414d2e7b65807ac0c6d637966a6b073e5d3bf3775c965a54de1a7115c2874\": container with ID starting with d6c414d2e7b65807ac0c6d637966a6b073e5d3bf3775c965a54de1a7115c2874 not found: ID does not exist" containerID="d6c414d2e7b65807ac0c6d637966a6b073e5d3bf3775c965a54de1a7115c2874"
Sep 09 21:03:40.759446 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.759356    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c414d2e7b65807ac0c6d637966a6b073e5d3bf3775c965a54de1a7115c2874"} err="failed to get container status \"d6c414d2e7b65807ac0c6d637966a6b073e5d3bf3775c965a54de1a7115c2874\": rpc error: code = NotFound desc = could not find container \"d6c414d2e7b65807ac0c6d637966a6b073e5d3bf3775c965a54de1a7115c2874\": container with ID starting with d6c414d2e7b65807ac0c6d637966a6b073e5d3bf3775c965a54de1a7115c2874 not found: ID does not exist"
Sep 09 21:03:40.759446 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.759378    2556 scope.go:117] "RemoveContainer" containerID="7e111343cec27d8917aaf405e28f0e3611107144aa40492fcebc6d8f84bfc8f6"
Sep 09 21:03:40.768157 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.768143    2556 scope.go:117] "RemoveContainer" containerID="7e111343cec27d8917aaf405e28f0e3611107144aa40492fcebc6d8f84bfc8f6"
Sep 09 21:03:40.768398 ip-10-0-135-117 kubenswrapper[2556]: E0909 21:03:40.768381    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e111343cec27d8917aaf405e28f0e3611107144aa40492fcebc6d8f84bfc8f6\": container with ID starting with 7e111343cec27d8917aaf405e28f0e3611107144aa40492fcebc6d8f84bfc8f6 not found: ID does not exist" containerID="7e111343cec27d8917aaf405e28f0e3611107144aa40492fcebc6d8f84bfc8f6"
Sep 09 21:03:40.768445 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.768405    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e111343cec27d8917aaf405e28f0e3611107144aa40492fcebc6d8f84bfc8f6"} err="failed to get container status \"7e111343cec27d8917aaf405e28f0e3611107144aa40492fcebc6d8f84bfc8f6\": rpc error: code = NotFound desc = could not find container \"7e111343cec27d8917aaf405e28f0e3611107144aa40492fcebc6d8f84bfc8f6\": container with ID starting with 7e111343cec27d8917aaf405e28f0e3611107144aa40492fcebc6d8f84bfc8f6 not found: ID does not exist"
Sep 09 21:03:40.768445 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.768419    2556 scope.go:117] "RemoveContainer" containerID="2ee2f7841d59af77792edaf2ec2fbdc6aa461167ac72c9296a2b45bb4142584b"
Sep 09 21:03:40.776639 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.776620    2556 scope.go:117] "RemoveContainer" containerID="2ee2f7841d59af77792edaf2ec2fbdc6aa461167ac72c9296a2b45bb4142584b"
Sep 09 21:03:40.776938 ip-10-0-135-117 kubenswrapper[2556]: E0909 21:03:40.776917    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ee2f7841d59af77792edaf2ec2fbdc6aa461167ac72c9296a2b45bb4142584b\": container with ID starting with 2ee2f7841d59af77792edaf2ec2fbdc6aa461167ac72c9296a2b45bb4142584b not found: ID does not exist" containerID="2ee2f7841d59af77792edaf2ec2fbdc6aa461167ac72c9296a2b45bb4142584b"
Sep 09 21:03:40.776992 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.776944    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee2f7841d59af77792edaf2ec2fbdc6aa461167ac72c9296a2b45bb4142584b"} err="failed to get container status \"2ee2f7841d59af77792edaf2ec2fbdc6aa461167ac72c9296a2b45bb4142584b\": rpc error: code = NotFound desc = could not find container \"2ee2f7841d59af77792edaf2ec2fbdc6aa461167ac72c9296a2b45bb4142584b\": container with ID starting with 2ee2f7841d59af77792edaf2ec2fbdc6aa461167ac72c9296a2b45bb4142584b not found: ID does not exist"
Sep 09 21:03:40.777560 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.777541    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"]
Sep 09 21:03:40.782025 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.782003    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/unconfigured-w4-933e9d-kserve-9dbd57777-g26cw"]
Sep 09 21:03:40.792448 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.792427    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"]
Sep 09 21:03:40.796104 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.796084    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/premium-w4-933e9d-kserve-7795b97f5f-fwqr8"]
Sep 09 21:03:40.806406 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.806386    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"]
Sep 09 21:03:40.809995 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.809976    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-933e9d/distinct-w4-933e9d-kserve-df9fb48cd-p4x77"]
Sep 09 21:03:40.892858 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.892798    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57b4976c-255a-4d22-884a-2492e4df333d" path="/var/lib/kubelet/pods/57b4976c-255a-4d22-884a-2492e4df333d/volumes"
Sep 09 21:03:40.893209 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.893195    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bc18bdc-1c2e-4414-8a04-30dead944112" path="/var/lib/kubelet/pods/6bc18bdc-1c2e-4414-8a04-30dead944112/volumes"
Sep 09 21:03:40.893554 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:40.893541    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccd2caf3-add9-49bc-a256-ac962f5b9a69" path="/var/lib/kubelet/pods/ccd2caf3-add9-49bc-a256-ac962f5b9a69/volumes"
Sep 09 21:03:41.195240 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:41.195158    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:03:57.003161 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:03:57.003127    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:04:11.092428 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:11.092396    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:04:23.405075 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:23.405033    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:04:29.302654 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:29.302621    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"]
Sep 09 21:04:35.390425 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:35.390334    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:04:44.321724 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:44.321683    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx" podUID="114f5b2b-f308-4c81-a3ca-f8ccb87c08cf" containerName="main" containerID="cri-o://a4a6b2ef16c33e828ec4c50db5f97c031c0cf955217369abb8879c89ec2e8d0f" gracePeriod=45
Sep 09 21:04:44.480228 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:44.480188    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx" podUID="114f5b2b-f308-4c81-a3ca-f8ccb87c08cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 09 21:04:44.564873 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:44.564850    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:04:44.621296 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:44.621226    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-dshm\") pod \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") "
Sep 09 21:04:44.621296 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:44.621279    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-model-cache\") pod \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") "
Sep 09 21:04:44.621480 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:44.621331    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86tpp\" (UniqueName: \"kubernetes.io/projected/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-kube-api-access-86tpp\") pod \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") "
Sep 09 21:04:44.621480 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:44.621371    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-tmp-dir\") pod \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") "
Sep 09 21:04:44.621480 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:44.621395    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-tls-certs\") pod \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") "
Sep 09 21:04:44.621480 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:44.621420    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-home\") pod \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\" (UID: \"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf\") "
Sep 09 21:04:44.621699 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:44.621551    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-model-cache" (OuterVolumeSpecName: "model-cache") pod "114f5b2b-f308-4c81-a3ca-f8ccb87c08cf" (UID: "114f5b2b-f308-4c81-a3ca-f8ccb87c08cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:44.621699 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:44.621664    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "114f5b2b-f308-4c81-a3ca-f8ccb87c08cf" (UID: "114f5b2b-f308-4c81-a3ca-f8ccb87c08cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:44.621809 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:44.621730    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-home" (OuterVolumeSpecName: "home") pod "114f5b2b-f308-4c81-a3ca-f8ccb87c08cf" (UID: "114f5b2b-f308-4c81-a3ca-f8ccb87c08cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:44.621809 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:44.621781    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-model-cache\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:04:44.621809 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:44.621802    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-tmp-dir\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:04:44.623571 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:44.623543    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "114f5b2b-f308-4c81-a3ca-f8ccb87c08cf" (UID: "114f5b2b-f308-4c81-a3ca-f8ccb87c08cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:04:44.623818 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:44.623798    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-dshm" (OuterVolumeSpecName: "dshm") pod "114f5b2b-f308-4c81-a3ca-f8ccb87c08cf" (UID: "114f5b2b-f308-4c81-a3ca-f8ccb87c08cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:44.623900 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:44.623820    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-kube-api-access-86tpp" (OuterVolumeSpecName: "kube-api-access-86tpp") pod "114f5b2b-f308-4c81-a3ca-f8ccb87c08cf" (UID: "114f5b2b-f308-4c81-a3ca-f8ccb87c08cf"). InnerVolumeSpecName "kube-api-access-86tpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:04:44.722533 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:44.722500    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-tls-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:04:44.722533 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:44.722529    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-home\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:04:44.722533 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:44.722538    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-dshm\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:04:44.722707 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:44.722549    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86tpp\" (UniqueName: \"kubernetes.io/projected/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf-kube-api-access-86tpp\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:04:45.112338 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:45.112308    2556 generic.go:356] "Generic (PLEG): container finished" podID="114f5b2b-f308-4c81-a3ca-f8ccb87c08cf" containerID="a4a6b2ef16c33e828ec4c50db5f97c031c0cf955217369abb8879c89ec2e8d0f" exitCode=0
Sep 09 21:04:45.112508 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:45.112347    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"
Sep 09 21:04:45.112508 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:45.112390    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx" event={"ID":"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf","Type":"ContainerDied","Data":"a4a6b2ef16c33e828ec4c50db5f97c031c0cf955217369abb8879c89ec2e8d0f"}
Sep 09 21:04:45.112508 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:45.112427    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx" event={"ID":"114f5b2b-f308-4c81-a3ca-f8ccb87c08cf","Type":"ContainerDied","Data":"eff43cc48e2624e2670a94e05a4943aa5d155940f9021ce73769adc182b6d6cc"}
Sep 09 21:04:45.112508 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:45.112445    2556 scope.go:117] "RemoveContainer" containerID="a4a6b2ef16c33e828ec4c50db5f97c031c0cf955217369abb8879c89ec2e8d0f"
Sep 09 21:04:45.123693 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:45.123667    2556 scope.go:117] "RemoveContainer" containerID="a4a6b2ef16c33e828ec4c50db5f97c031c0cf955217369abb8879c89ec2e8d0f"
Sep 09 21:04:45.123968 ip-10-0-135-117 kubenswrapper[2556]: E0909 21:04:45.123946    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4a6b2ef16c33e828ec4c50db5f97c031c0cf955217369abb8879c89ec2e8d0f\": container with ID starting with a4a6b2ef16c33e828ec4c50db5f97c031c0cf955217369abb8879c89ec2e8d0f not found: ID does not exist" containerID="a4a6b2ef16c33e828ec4c50db5f97c031c0cf955217369abb8879c89ec2e8d0f"
Sep 09 21:04:45.124041 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:45.123977    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4a6b2ef16c33e828ec4c50db5f97c031c0cf955217369abb8879c89ec2e8d0f"} err="failed to get container status \"a4a6b2ef16c33e828ec4c50db5f97c031c0cf955217369abb8879c89ec2e8d0f\": rpc error: code = NotFound desc = could not find container \"a4a6b2ef16c33e828ec4c50db5f97c031c0cf955217369abb8879c89ec2e8d0f\": container with ID starting with a4a6b2ef16c33e828ec4c50db5f97c031c0cf955217369abb8879c89ec2e8d0f not found: ID does not exist"
Sep 09 21:04:45.131745 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:45.131723    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"]
Sep 09 21:04:45.135828 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:45.135808    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cb82f2/sub-test-model-cb82f2-kserve-7688b596cc-qzpvx"]
Sep 09 21:04:45.705656 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:45.705625    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:04:46.895257 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:04:46.895217    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="114f5b2b-f308-4c81-a3ca-f8ccb87c08cf" path="/var/lib/kubelet/pods/114f5b2b-f308-4c81-a3ca-f8ccb87c08cf/volumes"
Sep 09 21:05:10.996095 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:10.996062    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:05:28.598041 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:28.597953    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:05:35.097194 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:35.097161    2556 scope.go:117] "RemoveContainer" containerID="fcc0578ccedd5c38851b5646e825e64d85495928395b7cf674b719272f4c51bb"
Sep 09 21:05:35.105275 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:35.105257    2556 scope.go:117] "RemoveContainer" containerID="b79c8b4a51e68ef350edf7d5f7ea1f7827d83dd43572519f520551f000b9e3d2"
Sep 09 21:05:37.895928 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:37.895878    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:05:47.899546 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:47.899510    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:05:51.052778 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.052748    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz"]
Sep 09 21:05:51.053160 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.053146    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccd2caf3-add9-49bc-a256-ac962f5b9a69" containerName="main"
Sep 09 21:05:51.053160 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.053161    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccd2caf3-add9-49bc-a256-ac962f5b9a69" containerName="main"
Sep 09 21:05:51.053242 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.053169    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57b4976c-255a-4d22-884a-2492e4df333d" containerName="main"
Sep 09 21:05:51.053242 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.053176    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="57b4976c-255a-4d22-884a-2492e4df333d" containerName="main"
Sep 09 21:05:51.053242 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.053191    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920ed061-dc28-4adf-b78d-d4aba5fb6b75" containerName="revoke-keys"
Sep 09 21:05:51.053242 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.053196    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="920ed061-dc28-4adf-b78d-d4aba5fb6b75" containerName="revoke-keys"
Sep 09 21:05:51.053242 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.053219    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="114f5b2b-f308-4c81-a3ca-f8ccb87c08cf" containerName="main"
Sep 09 21:05:51.053242 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.053224    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="114f5b2b-f308-4c81-a3ca-f8ccb87c08cf" containerName="main"
Sep 09 21:05:51.053242 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.053230    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bc18bdc-1c2e-4414-8a04-30dead944112" containerName="main"
Sep 09 21:05:51.053242 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.053235    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bc18bdc-1c2e-4414-8a04-30dead944112" containerName="main"
Sep 09 21:05:51.053478 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.053288    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="920ed061-dc28-4adf-b78d-d4aba5fb6b75" containerName="revoke-keys"
Sep 09 21:05:51.053478 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.053297    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="57b4976c-255a-4d22-884a-2492e4df333d" containerName="main"
Sep 09 21:05:51.053478 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.053303    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccd2caf3-add9-49bc-a256-ac962f5b9a69" containerName="main"
Sep 09 21:05:51.053478 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.053310    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="114f5b2b-f308-4c81-a3ca-f8ccb87c08cf" containerName="main"
Sep 09 21:05:51.053478 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.053316    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bc18bdc-1c2e-4414-8a04-30dead944112" containerName="main"
Sep 09 21:05:51.056646 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.056625    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz"
Sep 09 21:05:51.060531 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.060503    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-35e9ca\""
Sep 09 21:05:51.060643 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.060626    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-35e9ca-dockercfg-46k4n\""
Sep 09 21:05:51.068606 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.068584    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz"]
Sep 09 21:05:51.203058 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.203022    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz\" (UID: \"0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz"
Sep 09 21:05:51.203214 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.203076    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r9rf\" (UniqueName: \"kubernetes.io/projected/0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae-kube-api-access-9r9rf\") pod \"payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz\" (UID: \"0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz"
Sep 09 21:05:51.304165 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.304091    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz\" (UID: \"0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz"
Sep 09 21:05:51.304297 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.304164    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r9rf\" (UniqueName: \"kubernetes.io/projected/0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae-kube-api-access-9r9rf\") pod \"payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz\" (UID: \"0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz"
Sep 09 21:05:51.304736 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.304712    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz\" (UID: \"0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz"
Sep 09 21:05:51.313310 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.313289    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r9rf\" (UniqueName: \"kubernetes.io/projected/0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae-kube-api-access-9r9rf\") pod \"payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz\" (UID: \"0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz"
Sep 09 21:05:51.366425 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.366400    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz"
Sep 09 21:05:51.636868 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:51.636839    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz"]
Sep 09 21:05:51.639043 ip-10-0-135-117 kubenswrapper[2556]: W0909 21:05:51.639016    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b43fcc6_ca1a_4f5d_8139_60b47c3dc9ae.slice/crio-461fdbcf2d80b55175aece3b316b691c9b4e18dc59a1318981b9c07665ecc279 WatchSource:0}: Error finding container 461fdbcf2d80b55175aece3b316b691c9b4e18dc59a1318981b9c07665ecc279: Status 404 returned error can't find the container with id 461fdbcf2d80b55175aece3b316b691c9b4e18dc59a1318981b9c07665ecc279
Sep 09 21:05:52.476001 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:52.475967    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz" event={"ID":"0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae","Type":"ContainerStarted","Data":"994ec60260a0a4f4c7fd27f75dbba586a31e769e1762780efec8aecafbfd59cd"}
Sep 09 21:05:52.476410 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:52.476022    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz" event={"ID":"0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae","Type":"ContainerStarted","Data":"461fdbcf2d80b55175aece3b316b691c9b4e18dc59a1318981b9c07665ecc279"}
Sep 09 21:05:52.476410 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:52.476087    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz"
Sep 09 21:05:52.493570 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:05:52.493515    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz" podStartSLOduration=1.493500437 podStartE2EDuration="1.493500437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:05:52.491880234 +0000 UTC m=+1998.171840544" watchObservedRunningTime="2026-09-09 21:05:52.493500437 +0000 UTC m=+1998.173460752"
Sep 09 21:06:02.512889 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:02.512858    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:06:03.484288 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:03.484255    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz"
Sep 09 21:06:20.267352 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:20.267306    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb"]
Sep 09 21:06:20.271018 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:20.271002    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb"
Sep 09 21:06:20.274317 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:20.274276    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2ff038\""
Sep 09 21:06:20.274455 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:20.274322    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2ff038-dockercfg-59qtw\""
Sep 09 21:06:20.284508 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:20.284482    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb"]
Sep 09 21:06:20.347586 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:20.347560    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76224359-80b5-49b1-93d0-855d0b45ff9e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb\" (UID: \"76224359-80b5-49b1-93d0-855d0b45ff9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb"
Sep 09 21:06:20.347690 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:20.347677    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvdfk\" (UniqueName: \"kubernetes.io/projected/76224359-80b5-49b1-93d0-855d0b45ff9e-kube-api-access-pvdfk\") pod \"payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb\" (UID: \"76224359-80b5-49b1-93d0-855d0b45ff9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb"
Sep 09 21:06:20.449001 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:20.448961    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvdfk\" (UniqueName: \"kubernetes.io/projected/76224359-80b5-49b1-93d0-855d0b45ff9e-kube-api-access-pvdfk\") pod \"payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb\" (UID: \"76224359-80b5-49b1-93d0-855d0b45ff9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb"
Sep 09 21:06:20.449150 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:20.449052    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76224359-80b5-49b1-93d0-855d0b45ff9e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb\" (UID: \"76224359-80b5-49b1-93d0-855d0b45ff9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb"
Sep 09 21:06:20.449769 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:20.449751    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76224359-80b5-49b1-93d0-855d0b45ff9e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb\" (UID: \"76224359-80b5-49b1-93d0-855d0b45ff9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb"
Sep 09 21:06:20.457822 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:20.457789    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvdfk\" (UniqueName: \"kubernetes.io/projected/76224359-80b5-49b1-93d0-855d0b45ff9e-kube-api-access-pvdfk\") pod \"payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb\" (UID: \"76224359-80b5-49b1-93d0-855d0b45ff9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb"
Sep 09 21:06:20.581584 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:20.581522    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb"
Sep 09 21:06:20.859364 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:20.859337    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb"]
Sep 09 21:06:20.860969 ip-10-0-135-117 kubenswrapper[2556]: W0909 21:06:20.860938    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76224359_80b5_49b1_93d0_855d0b45ff9e.slice/crio-03b95bb131ae235f04b6c748363d601e954bac3b519e0a820e1207ebfadc25b1 WatchSource:0}: Error finding container 03b95bb131ae235f04b6c748363d601e954bac3b519e0a820e1207ebfadc25b1: Status 404 returned error can't find the container with id 03b95bb131ae235f04b6c748363d601e954bac3b519e0a820e1207ebfadc25b1
Sep 09 21:06:21.640420 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:21.640379    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb" event={"ID":"76224359-80b5-49b1-93d0-855d0b45ff9e","Type":"ContainerStarted","Data":"dbf2b107c38cbcca35b673bc1876d58feea75e4d1f795594572403efd18dbf50"}
Sep 09 21:06:21.640820 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:21.640444    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb" event={"ID":"76224359-80b5-49b1-93d0-855d0b45ff9e","Type":"ContainerStarted","Data":"03b95bb131ae235f04b6c748363d601e954bac3b519e0a820e1207ebfadc25b1"}
Sep 09 21:06:21.640820 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:21.640497    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb"
Sep 09 21:06:21.657986 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:21.657932    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb" podStartSLOduration=1.6578955130000002 podStartE2EDuration="1.657895513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:21.657568184 +0000 UTC m=+2027.337528494" watchObservedRunningTime="2026-09-09 21:06:21.657895513 +0000 UTC m=+2027.337855837"
Sep 09 21:06:26.000402 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:26.000355    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:06:32.648162 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:32.648132    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb"
Sep 09 21:06:35.131954 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:35.131920    2556 scope.go:117] "RemoveContainer" containerID="00c390eaf016954a21de85e340f2928c30ffd1f2bbb5ca79c8c06346fc922699"
Sep 09 21:06:35.140864 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:35.140843    2556 scope.go:117] "RemoveContainer" containerID="779e7751dc1c515878f09076200959441b5fe1e5e17178292d4e77f65ea86ed5"
Sep 09 21:06:35.148170 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:35.148153    2556 scope.go:117] "RemoveContainer" containerID="93d1933f71b582e88eddae12bef1dca40a6b57e247f05d79a63d967daa21c824"
Sep 09 21:06:35.155026 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:35.155009    2556 scope.go:117] "RemoveContainer" containerID="918d02628340a39b375280f4544eef39ccd8eafe67e20de4eb61ecdb260e92ee"
Sep 09 21:06:35.164882 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:35.164865    2556 scope.go:117] "RemoveContainer" containerID="0c24c98e42e88a97c9fa32958731d2709d69b638009a1824165a9a95980f118d"
Sep 09 21:06:35.172229 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:35.172213    2556 scope.go:117] "RemoveContainer" containerID="00a7c013076210d547d17d73a1f50131547df48965fc09f67fee59dfac916de0"
Sep 09 21:06:48.308369 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:48.308335    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb_76224359-80b5-49b1-93d0-855d0b45ff9e/payload-processing/0.log"
Sep 09 21:06:55.698662 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:06:55.698628    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:07:08.792415 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:08.792328    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:07:14.894580 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:14.894549    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb_76224359-80b5-49b1-93d0-855d0b45ff9e/payload-processing/0.log"
Sep 09 21:07:15.438310 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.438277    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"]
Sep 09 21:07:15.442295 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.442271    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.445089 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.445060    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-334ee7-openshift-default-dockercfg-9zqww\""
Sep 09 21:07:15.453812 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.453791    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"]
Sep 09 21:07:15.558314 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.558283    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-istio-envoy\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.558496 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.558337    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22l4z\" (UniqueName: \"kubernetes.io/projected/07b69822-5774-42cd-89cd-9915ba4e121a-kube-api-access-22l4z\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.558496 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.558393    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b69822-5774-42cd-89cd-9915ba4e121a-istio-token\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.558496 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.558424    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-istio-data\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.558659 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.558493    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b69822-5774-42cd-89cd-9915ba4e121a-istio-podinfo\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.558659 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.558519    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-workload-certs\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.558659 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.558553    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-workload-socket\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.558659 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.558608    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-credential-socket\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.558821 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.558668    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b69822-5774-42cd-89cd-9915ba4e121a-istiod-ca-cert\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.659102 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.659071    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-workload-socket\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.659102 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.659104    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-credential-socket\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.659314 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.659125    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b69822-5774-42cd-89cd-9915ba4e121a-istiod-ca-cert\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.659314 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.659153    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-istio-envoy\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.659314 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.659203    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22l4z\" (UniqueName: \"kubernetes.io/projected/07b69822-5774-42cd-89cd-9915ba4e121a-kube-api-access-22l4z\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.659314 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.659238    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b69822-5774-42cd-89cd-9915ba4e121a-istio-token\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.659314 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.659272    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-istio-data\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.659587 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.659327    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b69822-5774-42cd-89cd-9915ba4e121a-istio-podinfo\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.659587 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.659352    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-workload-certs\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.659587 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.659530    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-workload-socket\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.659587 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.659568    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-credential-socket\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.659743 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.659685    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-istio-data\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.659743 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.659721    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-workload-certs\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.660177 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.660156    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b69822-5774-42cd-89cd-9915ba4e121a-istiod-ca-cert\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.661594 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.661574    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-istio-envoy\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.661726 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.661712    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b69822-5774-42cd-89cd-9915ba4e121a-istio-podinfo\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.670329 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.670309    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22l4z\" (UniqueName: \"kubernetes.io/projected/07b69822-5774-42cd-89cd-9915ba4e121a-kube-api-access-22l4z\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.677735 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.677712    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b69822-5774-42cd-89cd-9915ba4e121a-istio-token\") pod \"e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") " pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:15.756870 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:15.756842    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:16.035143 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:16.035116    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"]
Sep 09 21:07:16.037399 ip-10-0-135-117 kubenswrapper[2556]: W0909 21:07:16.037372    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b69822_5774_42cd_89cd_9915ba4e121a.slice/crio-4cc32ece7e60cb51c90e53fe0cf28241bde958ec15d930a7f169df22400c1d7e WatchSource:0}: Error finding container 4cc32ece7e60cb51c90e53fe0cf28241bde958ec15d930a7f169df22400c1d7e: Status 404 returned error can't find the container with id 4cc32ece7e60cb51c90e53fe0cf28241bde958ec15d930a7f169df22400c1d7e
Sep 09 21:07:16.039505 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:16.039473    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:07:16.039594 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:16.039535    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:07:16.039594 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:16.039564    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:07:16.925536 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:16.925505    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv" event={"ID":"07b69822-5774-42cd-89cd-9915ba4e121a","Type":"ContainerStarted","Data":"7e66cb9a5223fb1f228715f2c5ae781978117a6f3abf9a8eb89914bbf014d645"}
Sep 09 21:07:16.925701 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:16.925550    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv" event={"ID":"07b69822-5774-42cd-89cd-9915ba4e121a","Type":"ContainerStarted","Data":"4cc32ece7e60cb51c90e53fe0cf28241bde958ec15d930a7f169df22400c1d7e"}
Sep 09 21:07:16.949783 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:16.949736    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv" podStartSLOduration=1.949722782 podStartE2EDuration="1.949722782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:07:16.948075037 +0000 UTC m=+2082.628035346" watchObservedRunningTime="2026-09-09 21:07:16.949722782 +0000 UTC m=+2082.629683093"
Sep 09 21:07:17.757435 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:17.757406    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:17.762477 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:17.762452    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:17.932797 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:17.932760    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:17.933493 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:17.933471    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:20.748850 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:20.748818    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t"]
Sep 09 21:07:20.752536 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:20.752516    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t"
Sep 09 21:07:20.755273 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:20.755253    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-334ee7-dockercfg-smm7v\""
Sep 09 21:07:20.755624 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:20.755611    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-334ee7\""
Sep 09 21:07:20.772859 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:20.772831    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t"]
Sep 09 21:07:20.905123 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:20.905088    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-676hp\" (UniqueName: \"kubernetes.io/projected/ae25dc55-9f33-4c69-a553-1ac2c0c354fd-kube-api-access-676hp\") pod \"payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t\" (UID: \"ae25dc55-9f33-4c69-a553-1ac2c0c354fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t"
Sep 09 21:07:20.905123 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:20.905128    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae25dc55-9f33-4c69-a553-1ac2c0c354fd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t\" (UID: \"ae25dc55-9f33-4c69-a553-1ac2c0c354fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t"
Sep 09 21:07:21.005705 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:21.005634    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-676hp\" (UniqueName: \"kubernetes.io/projected/ae25dc55-9f33-4c69-a553-1ac2c0c354fd-kube-api-access-676hp\") pod \"payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t\" (UID: \"ae25dc55-9f33-4c69-a553-1ac2c0c354fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t"
Sep 09 21:07:21.005705 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:21.005673    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae25dc55-9f33-4c69-a553-1ac2c0c354fd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t\" (UID: \"ae25dc55-9f33-4c69-a553-1ac2c0c354fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t"
Sep 09 21:07:21.006370 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:21.006351    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae25dc55-9f33-4c69-a553-1ac2c0c354fd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t\" (UID: \"ae25dc55-9f33-4c69-a553-1ac2c0c354fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t"
Sep 09 21:07:21.023434 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:21.023412    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-676hp\" (UniqueName: \"kubernetes.io/projected/ae25dc55-9f33-4c69-a553-1ac2c0c354fd-kube-api-access-676hp\") pod \"payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t\" (UID: \"ae25dc55-9f33-4c69-a553-1ac2c0c354fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t"
Sep 09 21:07:21.063257 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:21.063235    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t"
Sep 09 21:07:21.334533 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:21.334505    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t"]
Sep 09 21:07:21.336077 ip-10-0-135-117 kubenswrapper[2556]: W0909 21:07:21.336052    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae25dc55_9f33_4c69_a553_1ac2c0c354fd.slice/crio-b452e45e60c5f6fb3c1a1168ab0a5e8c11524116245f102fab1b0a637fd6f4bb WatchSource:0}: Error finding container b452e45e60c5f6fb3c1a1168ab0a5e8c11524116245f102fab1b0a637fd6f4bb: Status 404 returned error can't find the container with id b452e45e60c5f6fb3c1a1168ab0a5e8c11524116245f102fab1b0a637fd6f4bb
Sep 09 21:07:21.952812 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:21.952776    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t" event={"ID":"ae25dc55-9f33-4c69-a553-1ac2c0c354fd","Type":"ContainerStarted","Data":"5f9a10513ff48328893682e71588b656ae1223ec5f0e2ec94e77830cede39bb4"}
Sep 09 21:07:21.953214 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:21.952835    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t"
Sep 09 21:07:21.953214 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:21.952853    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t" event={"ID":"ae25dc55-9f33-4c69-a553-1ac2c0c354fd","Type":"ContainerStarted","Data":"b452e45e60c5f6fb3c1a1168ab0a5e8c11524116245f102fab1b0a637fd6f4bb"}
Sep 09 21:07:21.971537 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:21.971492    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t" podStartSLOduration=1.971476793 podStartE2EDuration="1.971476793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:07:21.969817327 +0000 UTC m=+2087.649777656" watchObservedRunningTime="2026-09-09 21:07:21.971476793 +0000 UTC m=+2087.651437093"
Sep 09 21:07:27.802040 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:27.802004    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:07:32.959732 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:32.959702    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t"
Sep 09 21:07:38.041591 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:38.041561    2556 generic.go:356] "Generic (PLEG): container finished" podID="07b69822-5774-42cd-89cd-9915ba4e121a" containerID="7e66cb9a5223fb1f228715f2c5ae781978117a6f3abf9a8eb89914bbf014d645" exitCode=0
Sep 09 21:07:38.042104 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:38.041624    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv" event={"ID":"07b69822-5774-42cd-89cd-9915ba4e121a","Type":"ContainerDied","Data":"7e66cb9a5223fb1f228715f2c5ae781978117a6f3abf9a8eb89914bbf014d645"}
Sep 09 21:07:38.042177 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:38.042144    2556 scope.go:117] "RemoveContainer" containerID="7e66cb9a5223fb1f228715f2c5ae781978117a6f3abf9a8eb89914bbf014d645"
Sep 09 21:07:38.043255 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:38.043223    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:07:38.043363 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:38.043311    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:07:38.043363 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:38.043356    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:07:39.048626 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:39.048586    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv" event={"ID":"07b69822-5774-42cd-89cd-9915ba4e121a","Type":"ContainerStarted","Data":"9c20f6dcc1575b7910d7aca8b561d83ee5a759629127852d9b11707c2ca6e5ce"}
Sep 09 21:07:39.757758 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:39.757719    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:39.758023 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:39.757997    2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 09 21:07:39.758081 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:39.758061    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv" podUID="07b69822-5774-42cd-89cd-9915ba4e121a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 09 21:07:40.757766 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:40.757727    2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 09 21:07:40.758214 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:40.757782    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv" podUID="07b69822-5774-42cd-89cd-9915ba4e121a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 09 21:07:41.758070 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:41.758034    2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 09 21:07:41.758464 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:41.758086    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv" podUID="07b69822-5774-42cd-89cd-9915ba4e121a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 09 21:07:42.757916 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:42.757877    2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 09 21:07:42.758097 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:42.757966    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv" podUID="07b69822-5774-42cd-89cd-9915ba4e121a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 09 21:07:43.757192 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:43.757152    2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 09 21:07:43.757374 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:43.757219    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv" podUID="07b69822-5774-42cd-89cd-9915ba4e121a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 09 21:07:44.440104 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:44.440069    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-334ee7-9mvjx"]
Sep 09 21:07:44.443756 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:44.443739    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-334ee7-9mvjx"
Sep 09 21:07:44.446492 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:44.446469    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 21:07:44.446611 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:44.446471    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsks9\""
Sep 09 21:07:44.451399 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:44.451379    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-334ee7-9mvjx"]
Sep 09 21:07:44.533094 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:44.533051    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7291a385-ecbc-4d91-9422-2b96d20c8382-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-334ee7-9mvjx\" (UID: \"7291a385-ecbc-4d91-9422-2b96d20c8382\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-334ee7-9mvjx"
Sep 09 21:07:44.633924 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:44.633873    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7291a385-ecbc-4d91-9422-2b96d20c8382-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-334ee7-9mvjx\" (UID: \"7291a385-ecbc-4d91-9422-2b96d20c8382\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-334ee7-9mvjx"
Sep 09 21:07:44.634958 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:44.634899    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7291a385-ecbc-4d91-9422-2b96d20c8382-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-334ee7-9mvjx\" (UID: \"7291a385-ecbc-4d91-9422-2b96d20c8382\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-334ee7-9mvjx"
Sep 09 21:07:44.755094 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:44.755042    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-334ee7-9mvjx"
Sep 09 21:07:44.803059 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:44.802968    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:44.803787 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:44.803761    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:44.804611 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:44.804587    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:07:45.036120 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:45.036093    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-334ee7-9mvjx"]
Sep 09 21:07:45.038248 ip-10-0-135-117 kubenswrapper[2556]: W0909 21:07:45.038219    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7291a385_ecbc_4d91_9422_2b96d20c8382.slice/crio-6e447ab45236ffe04b6ee2f08fda98272486e2ff402cff9c4da8b297e5902601 WatchSource:0}: Error finding container 6e447ab45236ffe04b6ee2f08fda98272486e2ff402cff9c4da8b297e5902601: Status 404 returned error can't find the container with id 6e447ab45236ffe04b6ee2f08fda98272486e2ff402cff9c4da8b297e5902601
Sep 09 21:07:45.082784 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:45.082761    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-334ee7-9mvjx" event={"ID":"7291a385-ecbc-4d91-9422-2b96d20c8382","Type":"ContainerStarted","Data":"6e447ab45236ffe04b6ee2f08fda98272486e2ff402cff9c4da8b297e5902601"}
Sep 09 21:07:46.089130 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:46.089098    2556 generic.go:356] "Generic (PLEG): container finished" podID="7291a385-ecbc-4d91-9422-2b96d20c8382" containerID="2dc81f8952bc5ce52951bc3000d8faaef17810624429944f29903c8f4d53b06e" exitCode=0
Sep 09 21:07:46.089499 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:46.089156    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-334ee7-9mvjx" event={"ID":"7291a385-ecbc-4d91-9422-2b96d20c8382","Type":"ContainerDied","Data":"2dc81f8952bc5ce52951bc3000d8faaef17810624429944f29903c8f4d53b06e"}
Sep 09 21:07:47.221945 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:47.221926    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-334ee7-9mvjx"
Sep 09 21:07:47.360131 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:47.360057    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7291a385-ecbc-4d91-9422-2b96d20c8382-maas-api-service-ca\") pod \"7291a385-ecbc-4d91-9422-2b96d20c8382\" (UID: \"7291a385-ecbc-4d91-9422-2b96d20c8382\") "
Sep 09 21:07:47.360439 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:47.360409    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7291a385-ecbc-4d91-9422-2b96d20c8382-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7291a385-ecbc-4d91-9422-2b96d20c8382" (UID: "7291a385-ecbc-4d91-9422-2b96d20c8382"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:07:47.461637 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:47.461605    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7291a385-ecbc-4d91-9422-2b96d20c8382-maas-api-service-ca\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:07:48.102078 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:48.102051    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-334ee7-9mvjx"
Sep 09 21:07:48.102235 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:48.102051    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-334ee7-9mvjx" event={"ID":"7291a385-ecbc-4d91-9422-2b96d20c8382","Type":"ContainerDied","Data":"6e447ab45236ffe04b6ee2f08fda98272486e2ff402cff9c4da8b297e5902601"}
Sep 09 21:07:48.102235 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:48.102178    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e447ab45236ffe04b6ee2f08fda98272486e2ff402cff9c4da8b297e5902601"
Sep 09 21:07:50.665772 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:50.665738    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t"]
Sep 09 21:07:50.666269 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:50.666002    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t" podUID="ae25dc55-9f33-4c69-a553-1ac2c0c354fd" containerName="payload-processing" containerID="cri-o://5f9a10513ff48328893682e71588b656ae1223ec5f0e2ec94e77830cede39bb4" gracePeriod=30
Sep 09 21:07:50.846544 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:50.846512    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-334ee7-9mvjx"]
Sep 09 21:07:50.849833 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:50.849807    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-334ee7-9mvjx"]
Sep 09 21:07:50.892736 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:50.892709    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7291a385-ecbc-4d91-9422-2b96d20c8382" path="/var/lib/kubelet/pods/7291a385-ecbc-4d91-9422-2b96d20c8382/volumes"
Sep 09 21:07:50.919168 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:50.919118    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t"
Sep 09 21:07:50.989348 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:50.989325    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae25dc55-9f33-4c69-a553-1ac2c0c354fd-plugins-config-volume\") pod \"ae25dc55-9f33-4c69-a553-1ac2c0c354fd\" (UID: \"ae25dc55-9f33-4c69-a553-1ac2c0c354fd\") "
Sep 09 21:07:50.989489 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:50.989380    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-676hp\" (UniqueName: \"kubernetes.io/projected/ae25dc55-9f33-4c69-a553-1ac2c0c354fd-kube-api-access-676hp\") pod \"ae25dc55-9f33-4c69-a553-1ac2c0c354fd\" (UID: \"ae25dc55-9f33-4c69-a553-1ac2c0c354fd\") "
Sep 09 21:07:50.989689 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:50.989666    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae25dc55-9f33-4c69-a553-1ac2c0c354fd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ae25dc55-9f33-4c69-a553-1ac2c0c354fd" (UID: "ae25dc55-9f33-4c69-a553-1ac2c0c354fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:07:50.989796 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:50.989779    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae25dc55-9f33-4c69-a553-1ac2c0c354fd-plugins-config-volume\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:07:50.991442 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:50.991410    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae25dc55-9f33-4c69-a553-1ac2c0c354fd-kube-api-access-676hp" (OuterVolumeSpecName: "kube-api-access-676hp") pod "ae25dc55-9f33-4c69-a553-1ac2c0c354fd" (UID: "ae25dc55-9f33-4c69-a553-1ac2c0c354fd"). InnerVolumeSpecName "kube-api-access-676hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:07:51.090714 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:51.090686    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-676hp\" (UniqueName: \"kubernetes.io/projected/ae25dc55-9f33-4c69-a553-1ac2c0c354fd-kube-api-access-676hp\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:07:51.118888 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:51.118860    2556 generic.go:356] "Generic (PLEG): container finished" podID="ae25dc55-9f33-4c69-a553-1ac2c0c354fd" containerID="5f9a10513ff48328893682e71588b656ae1223ec5f0e2ec94e77830cede39bb4" exitCode=0
Sep 09 21:07:51.119040 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:51.118901    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t"
Sep 09 21:07:51.119040 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:51.118949    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t" event={"ID":"ae25dc55-9f33-4c69-a553-1ac2c0c354fd","Type":"ContainerDied","Data":"5f9a10513ff48328893682e71588b656ae1223ec5f0e2ec94e77830cede39bb4"}
Sep 09 21:07:51.119040 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:51.119001    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t" event={"ID":"ae25dc55-9f33-4c69-a553-1ac2c0c354fd","Type":"ContainerDied","Data":"b452e45e60c5f6fb3c1a1168ab0a5e8c11524116245f102fab1b0a637fd6f4bb"}
Sep 09 21:07:51.119040 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:51.119022    2556 scope.go:117] "RemoveContainer" containerID="5f9a10513ff48328893682e71588b656ae1223ec5f0e2ec94e77830cede39bb4"
Sep 09 21:07:51.131406 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:51.131381    2556 scope.go:117] "RemoveContainer" containerID="5f9a10513ff48328893682e71588b656ae1223ec5f0e2ec94e77830cede39bb4"
Sep 09 21:07:51.131872 ip-10-0-135-117 kubenswrapper[2556]: E0909 21:07:51.131845    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f9a10513ff48328893682e71588b656ae1223ec5f0e2ec94e77830cede39bb4\": container with ID starting with 5f9a10513ff48328893682e71588b656ae1223ec5f0e2ec94e77830cede39bb4 not found: ID does not exist" containerID="5f9a10513ff48328893682e71588b656ae1223ec5f0e2ec94e77830cede39bb4"
Sep 09 21:07:51.131993 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:51.131881    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f9a10513ff48328893682e71588b656ae1223ec5f0e2ec94e77830cede39bb4"} err="failed to get container status \"5f9a10513ff48328893682e71588b656ae1223ec5f0e2ec94e77830cede39bb4\": rpc error: code = NotFound desc = could not find container \"5f9a10513ff48328893682e71588b656ae1223ec5f0e2ec94e77830cede39bb4\": container with ID starting with 5f9a10513ff48328893682e71588b656ae1223ec5f0e2ec94e77830cede39bb4 not found: ID does not exist"
Sep 09 21:07:51.145033 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:51.145009    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t"]
Sep 09 21:07:51.148553 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:51.148532    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-334ee7-75b8bbf789-nls7t"]
Sep 09 21:07:52.892595 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:52.892563    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae25dc55-9f33-4c69-a553-1ac2c0c354fd" path="/var/lib/kubelet/pods/ae25dc55-9f33-4c69-a553-1ac2c0c354fd/volumes"
Sep 09 21:07:55.192995 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:55.192960    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:07:57.689285 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:57.689253    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"]
Sep 09 21:07:57.689695 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:57.689482    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv" podUID="07b69822-5774-42cd-89cd-9915ba4e121a" containerName="istio-proxy" containerID="cri-o://9c20f6dcc1575b7910d7aca8b561d83ee5a759629127852d9b11707c2ca6e5ce" gracePeriod=30
Sep 09 21:07:59.806698 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:59.806656    2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 21:07:59.807139 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:07:59.806733    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv" podUID="07b69822-5774-42cd-89cd-9915ba4e121a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:08:01.187725 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:01.187691    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-35e9ca-vhc2k"]
Sep 09 21:08:01.188192 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:01.188174    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7291a385-ecbc-4d91-9422-2b96d20c8382" containerName="revoke-keys"
Sep 09 21:08:01.188265 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:01.188193    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7291a385-ecbc-4d91-9422-2b96d20c8382" containerName="revoke-keys"
Sep 09 21:08:01.188265 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:01.188208    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae25dc55-9f33-4c69-a553-1ac2c0c354fd" containerName="payload-processing"
Sep 09 21:08:01.188265 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:01.188215    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae25dc55-9f33-4c69-a553-1ac2c0c354fd" containerName="payload-processing"
Sep 09 21:08:01.188368 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:01.188275    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae25dc55-9f33-4c69-a553-1ac2c0c354fd" containerName="payload-processing"
Sep 09 21:08:01.188368 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:01.188284    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7291a385-ecbc-4d91-9422-2b96d20c8382" containerName="revoke-keys"
Sep 09 21:08:01.192900 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:01.192884    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-35e9ca-vhc2k"
Sep 09 21:08:01.195758 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:01.195734    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsks9\""
Sep 09 21:08:01.195882 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:01.195743    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 21:08:01.199679 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:01.199654    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-35e9ca-vhc2k"]
Sep 09 21:08:01.279478 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:01.279442    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ac943b-f4b4-4ba6-ad24-627a192232b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-35e9ca-vhc2k\" (UID: \"d2ac943b-f4b4-4ba6-ad24-627a192232b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-35e9ca-vhc2k"
Sep 09 21:08:01.380308 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:01.380270    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ac943b-f4b4-4ba6-ad24-627a192232b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-35e9ca-vhc2k\" (UID: \"d2ac943b-f4b4-4ba6-ad24-627a192232b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-35e9ca-vhc2k"
Sep 09 21:08:01.380931 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:01.380886    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ac943b-f4b4-4ba6-ad24-627a192232b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-35e9ca-vhc2k\" (UID: \"d2ac943b-f4b4-4ba6-ad24-627a192232b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-35e9ca-vhc2k"
Sep 09 21:08:01.504194 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:01.504169    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-35e9ca-vhc2k"
Sep 09 21:08:01.773942 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:01.773898    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-35e9ca-vhc2k"]
Sep 09 21:08:01.774759 ip-10-0-135-117 kubenswrapper[2556]: W0909 21:08:01.774737    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ac943b_f4b4_4ba6_ad24_627a192232b0.slice/crio-bdc8224b715d19ef408ccbe77d7bf794a8d3134bb63ae5f1a3e91e1e2037a49d WatchSource:0}: Error finding container bdc8224b715d19ef408ccbe77d7bf794a8d3134bb63ae5f1a3e91e1e2037a49d: Status 404 returned error can't find the container with id bdc8224b715d19ef408ccbe77d7bf794a8d3134bb63ae5f1a3e91e1e2037a49d
Sep 09 21:08:02.177746 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:02.177714    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-35e9ca-vhc2k" event={"ID":"d2ac943b-f4b4-4ba6-ad24-627a192232b0","Type":"ContainerStarted","Data":"a0124428d202cafc2a4f90be78f2a8c2193d029d17b183cb92c33c14880cd4db"}
Sep 09 21:08:02.177856 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:02.177767    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-35e9ca-vhc2k" event={"ID":"d2ac943b-f4b4-4ba6-ad24-627a192232b0","Type":"ContainerStarted","Data":"bdc8224b715d19ef408ccbe77d7bf794a8d3134bb63ae5f1a3e91e1e2037a49d"}
Sep 09 21:08:02.209178 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:02.209132    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-35e9ca-vhc2k" podStartSLOduration=1.209117927 podStartE2EDuration="1.209117927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:08:02.205677295 +0000 UTC m=+2127.885637604" watchObservedRunningTime="2026-09-09 21:08:02.209117927 +0000 UTC m=+2127.889078236"
Sep 09 21:08:02.941764 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:02.941740    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:08:03.094794 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.094719    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b69822-5774-42cd-89cd-9915ba4e121a-istiod-ca-cert\") pod \"07b69822-5774-42cd-89cd-9915ba4e121a\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") "
Sep 09 21:08:03.094794 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.094760    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-workload-socket\") pod \"07b69822-5774-42cd-89cd-9915ba4e121a\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") "
Sep 09 21:08:03.094794 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.094791    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-istio-envoy\") pod \"07b69822-5774-42cd-89cd-9915ba4e121a\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") "
Sep 09 21:08:03.095094 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.094826    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b69822-5774-42cd-89cd-9915ba4e121a-istio-token\") pod \"07b69822-5774-42cd-89cd-9915ba4e121a\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") "
Sep 09 21:08:03.095094 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.094867    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22l4z\" (UniqueName: \"kubernetes.io/projected/07b69822-5774-42cd-89cd-9915ba4e121a-kube-api-access-22l4z\") pod \"07b69822-5774-42cd-89cd-9915ba4e121a\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") "
Sep 09 21:08:03.095094 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.094894    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-istio-data\") pod \"07b69822-5774-42cd-89cd-9915ba4e121a\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") "
Sep 09 21:08:03.095094 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.094961    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-workload-certs\") pod \"07b69822-5774-42cd-89cd-9915ba4e121a\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") "
Sep 09 21:08:03.095094 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.095035    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b69822-5774-42cd-89cd-9915ba4e121a-istio-podinfo\") pod \"07b69822-5774-42cd-89cd-9915ba4e121a\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") "
Sep 09 21:08:03.095094 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.095080    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-credential-socket\") pod \"07b69822-5774-42cd-89cd-9915ba4e121a\" (UID: \"07b69822-5774-42cd-89cd-9915ba4e121a\") "
Sep 09 21:08:03.095421 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.095097    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "07b69822-5774-42cd-89cd-9915ba4e121a" (UID: "07b69822-5774-42cd-89cd-9915ba4e121a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:03.095421 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.095158    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b69822-5774-42cd-89cd-9915ba4e121a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "07b69822-5774-42cd-89cd-9915ba4e121a" (UID: "07b69822-5774-42cd-89cd-9915ba4e121a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:08:03.095421 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.095412    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b69822-5774-42cd-89cd-9915ba4e121a-istiod-ca-cert\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:08:03.095588 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.095430    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-workload-socket\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:08:03.095588 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.095414    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "07b69822-5774-42cd-89cd-9915ba4e121a" (UID: "07b69822-5774-42cd-89cd-9915ba4e121a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:03.095588 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.095470    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "07b69822-5774-42cd-89cd-9915ba4e121a" (UID: "07b69822-5774-42cd-89cd-9915ba4e121a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:03.095588 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.095533    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-istio-data" (OuterVolumeSpecName: "istio-data") pod "07b69822-5774-42cd-89cd-9915ba4e121a" (UID: "07b69822-5774-42cd-89cd-9915ba4e121a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:03.097040 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.097010    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "07b69822-5774-42cd-89cd-9915ba4e121a" (UID: "07b69822-5774-42cd-89cd-9915ba4e121a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:03.097156 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.097089    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07b69822-5774-42cd-89cd-9915ba4e121a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "07b69822-5774-42cd-89cd-9915ba4e121a" (UID: "07b69822-5774-42cd-89cd-9915ba4e121a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:08:03.097470 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.097450    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b69822-5774-42cd-89cd-9915ba4e121a-kube-api-access-22l4z" (OuterVolumeSpecName: "kube-api-access-22l4z") pod "07b69822-5774-42cd-89cd-9915ba4e121a" (UID: "07b69822-5774-42cd-89cd-9915ba4e121a"). InnerVolumeSpecName "kube-api-access-22l4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:08:03.097470 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.097463    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b69822-5774-42cd-89cd-9915ba4e121a-istio-token" (OuterVolumeSpecName: "istio-token") pod "07b69822-5774-42cd-89cd-9915ba4e121a" (UID: "07b69822-5774-42cd-89cd-9915ba4e121a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:08:03.184169 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.184142    2556 generic.go:356] "Generic (PLEG): container finished" podID="07b69822-5774-42cd-89cd-9915ba4e121a" containerID="9c20f6dcc1575b7910d7aca8b561d83ee5a759629127852d9b11707c2ca6e5ce" exitCode=0
Sep 09 21:08:03.184307 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.184193    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"
Sep 09 21:08:03.184307 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.184224    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv" event={"ID":"07b69822-5774-42cd-89cd-9915ba4e121a","Type":"ContainerDied","Data":"9c20f6dcc1575b7910d7aca8b561d83ee5a759629127852d9b11707c2ca6e5ce"}
Sep 09 21:08:03.184307 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.184283    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv" event={"ID":"07b69822-5774-42cd-89cd-9915ba4e121a","Type":"ContainerDied","Data":"4cc32ece7e60cb51c90e53fe0cf28241bde958ec15d930a7f169df22400c1d7e"}
Sep 09 21:08:03.184475 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.184317    2556 scope.go:117] "RemoveContainer" containerID="9c20f6dcc1575b7910d7aca8b561d83ee5a759629127852d9b11707c2ca6e5ce"
Sep 09 21:08:03.186132 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.186107    2556 generic.go:356] "Generic (PLEG): container finished" podID="d2ac943b-f4b4-4ba6-ad24-627a192232b0" containerID="a0124428d202cafc2a4f90be78f2a8c2193d029d17b183cb92c33c14880cd4db" exitCode=0
Sep 09 21:08:03.186257 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.186198    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-35e9ca-vhc2k" event={"ID":"d2ac943b-f4b4-4ba6-ad24-627a192232b0","Type":"ContainerDied","Data":"a0124428d202cafc2a4f90be78f2a8c2193d029d17b183cb92c33c14880cd4db"}
Sep 09 21:08:03.198508 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.198089    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-istio-envoy\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:08:03.198508 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.198120    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b69822-5774-42cd-89cd-9915ba4e121a-istio-token\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:08:03.198508 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.198141    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22l4z\" (UniqueName: \"kubernetes.io/projected/07b69822-5774-42cd-89cd-9915ba4e121a-kube-api-access-22l4z\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:08:03.198508 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.198155    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-istio-data\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:08:03.198508 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.198170    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-workload-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:08:03.198508 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.198184    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b69822-5774-42cd-89cd-9915ba4e121a-istio-podinfo\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:08:03.198508 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.198201    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b69822-5774-42cd-89cd-9915ba4e121a-credential-socket\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:08:03.199585 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.199561    2556 scope.go:117] "RemoveContainer" containerID="7e66cb9a5223fb1f228715f2c5ae781978117a6f3abf9a8eb89914bbf014d645"
Sep 09 21:08:03.207881 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.207862    2556 scope.go:117] "RemoveContainer" containerID="9c20f6dcc1575b7910d7aca8b561d83ee5a759629127852d9b11707c2ca6e5ce"
Sep 09 21:08:03.208160 ip-10-0-135-117 kubenswrapper[2556]: E0909 21:08:03.208141    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c20f6dcc1575b7910d7aca8b561d83ee5a759629127852d9b11707c2ca6e5ce\": container with ID starting with 9c20f6dcc1575b7910d7aca8b561d83ee5a759629127852d9b11707c2ca6e5ce not found: ID does not exist" containerID="9c20f6dcc1575b7910d7aca8b561d83ee5a759629127852d9b11707c2ca6e5ce"
Sep 09 21:08:03.208218 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.208168    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c20f6dcc1575b7910d7aca8b561d83ee5a759629127852d9b11707c2ca6e5ce"} err="failed to get container status \"9c20f6dcc1575b7910d7aca8b561d83ee5a759629127852d9b11707c2ca6e5ce\": rpc error: code = NotFound desc = could not find container \"9c20f6dcc1575b7910d7aca8b561d83ee5a759629127852d9b11707c2ca6e5ce\": container with ID starting with 9c20f6dcc1575b7910d7aca8b561d83ee5a759629127852d9b11707c2ca6e5ce not found: ID does not exist"
Sep 09 21:08:03.208218 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.208187    2556 scope.go:117] "RemoveContainer" containerID="7e66cb9a5223fb1f228715f2c5ae781978117a6f3abf9a8eb89914bbf014d645"
Sep 09 21:08:03.208407 ip-10-0-135-117 kubenswrapper[2556]: E0909 21:08:03.208392    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e66cb9a5223fb1f228715f2c5ae781978117a6f3abf9a8eb89914bbf014d645\": container with ID starting with 7e66cb9a5223fb1f228715f2c5ae781978117a6f3abf9a8eb89914bbf014d645 not found: ID does not exist" containerID="7e66cb9a5223fb1f228715f2c5ae781978117a6f3abf9a8eb89914bbf014d645"
Sep 09 21:08:03.208448 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.208409    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e66cb9a5223fb1f228715f2c5ae781978117a6f3abf9a8eb89914bbf014d645"} err="failed to get container status \"7e66cb9a5223fb1f228715f2c5ae781978117a6f3abf9a8eb89914bbf014d645\": rpc error: code = NotFound desc = could not find container \"7e66cb9a5223fb1f228715f2c5ae781978117a6f3abf9a8eb89914bbf014d645\": container with ID starting with 7e66cb9a5223fb1f228715f2c5ae781978117a6f3abf9a8eb89914bbf014d645 not found: ID does not exist"
Sep 09 21:08:03.235350 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.235330    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"]
Sep 09 21:08:03.238009 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:03.237985    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-334ee7-openshift-default-7bb6dd99f4-426mv"]
Sep 09 21:08:04.328280 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:04.328254    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-35e9ca-vhc2k"
Sep 09 21:08:04.408665 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:04.408639    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ac943b-f4b4-4ba6-ad24-627a192232b0-maas-api-service-ca\") pod \"d2ac943b-f4b4-4ba6-ad24-627a192232b0\" (UID: \"d2ac943b-f4b4-4ba6-ad24-627a192232b0\") "
Sep 09 21:08:04.409001 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:04.408980    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ac943b-f4b4-4ba6-ad24-627a192232b0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d2ac943b-f4b4-4ba6-ad24-627a192232b0" (UID: "d2ac943b-f4b4-4ba6-ad24-627a192232b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:08:04.510207 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:04.510178    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ac943b-f4b4-4ba6-ad24-627a192232b0-maas-api-service-ca\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:08:04.892599 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:04.892529    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07b69822-5774-42cd-89cd-9915ba4e121a" path="/var/lib/kubelet/pods/07b69822-5774-42cd-89cd-9915ba4e121a/volumes"
Sep 09 21:08:05.198247 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:05.198164    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-35e9ca-vhc2k"
Sep 09 21:08:05.198247 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:05.198180    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-35e9ca-vhc2k" event={"ID":"d2ac943b-f4b4-4ba6-ad24-627a192232b0","Type":"ContainerDied","Data":"bdc8224b715d19ef408ccbe77d7bf794a8d3134bb63ae5f1a3e91e1e2037a49d"}
Sep 09 21:08:05.198247 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:05.198223    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdc8224b715d19ef408ccbe77d7bf794a8d3134bb63ae5f1a3e91e1e2037a49d"
Sep 09 21:08:14.303930 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:14.303886    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:08:16.740749 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:16.740718    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz"]
Sep 09 21:08:16.741237 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:16.741006    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz" podUID="0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae" containerName="payload-pre-processing" containerID="cri-o://994ec60260a0a4f4c7fd27f75dbba586a31e769e1762780efec8aecafbfd59cd" gracePeriod=30
Sep 09 21:08:16.950568 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:16.950535    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-35e9ca-vhc2k"]
Sep 09 21:08:16.953404 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:16.953380    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-35e9ca-vhc2k"]
Sep 09 21:08:16.999214 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:16.999160    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz"
Sep 09 21:08:17.118071 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:17.118034    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae-plugins-config-volume\") pod \"0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae\" (UID: \"0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae\") "
Sep 09 21:08:17.118275 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:17.118080    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r9rf\" (UniqueName: \"kubernetes.io/projected/0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae-kube-api-access-9r9rf\") pod \"0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae\" (UID: \"0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae\") "
Sep 09 21:08:17.118477 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:17.118450    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae" (UID: "0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:08:17.120179 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:17.120157    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae-kube-api-access-9r9rf" (OuterVolumeSpecName: "kube-api-access-9r9rf") pod "0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae" (UID: "0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae"). InnerVolumeSpecName "kube-api-access-9r9rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:08:17.219291 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:17.219268    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae-plugins-config-volume\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:08:17.219291 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:17.219291    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r9rf\" (UniqueName: \"kubernetes.io/projected/0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae-kube-api-access-9r9rf\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:08:17.266633 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:17.266567    2556 generic.go:356] "Generic (PLEG): container finished" podID="0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae" containerID="994ec60260a0a4f4c7fd27f75dbba586a31e769e1762780efec8aecafbfd59cd" exitCode=0
Sep 09 21:08:17.266633 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:17.266613    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz"
Sep 09 21:08:17.266764 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:17.266657    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz" event={"ID":"0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae","Type":"ContainerDied","Data":"994ec60260a0a4f4c7fd27f75dbba586a31e769e1762780efec8aecafbfd59cd"}
Sep 09 21:08:17.266764 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:17.266705    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz" event={"ID":"0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae","Type":"ContainerDied","Data":"461fdbcf2d80b55175aece3b316b691c9b4e18dc59a1318981b9c07665ecc279"}
Sep 09 21:08:17.266764 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:17.266730    2556 scope.go:117] "RemoveContainer" containerID="994ec60260a0a4f4c7fd27f75dbba586a31e769e1762780efec8aecafbfd59cd"
Sep 09 21:08:17.276603 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:17.276588    2556 scope.go:117] "RemoveContainer" containerID="994ec60260a0a4f4c7fd27f75dbba586a31e769e1762780efec8aecafbfd59cd"
Sep 09 21:08:17.276850 ip-10-0-135-117 kubenswrapper[2556]: E0909 21:08:17.276833    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"994ec60260a0a4f4c7fd27f75dbba586a31e769e1762780efec8aecafbfd59cd\": container with ID starting with 994ec60260a0a4f4c7fd27f75dbba586a31e769e1762780efec8aecafbfd59cd not found: ID does not exist" containerID="994ec60260a0a4f4c7fd27f75dbba586a31e769e1762780efec8aecafbfd59cd"
Sep 09 21:08:17.276900 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:17.276857    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"994ec60260a0a4f4c7fd27f75dbba586a31e769e1762780efec8aecafbfd59cd"} err="failed to get container status \"994ec60260a0a4f4c7fd27f75dbba586a31e769e1762780efec8aecafbfd59cd\": rpc error: code = NotFound desc = could not find container \"994ec60260a0a4f4c7fd27f75dbba586a31e769e1762780efec8aecafbfd59cd\": container with ID starting with 994ec60260a0a4f4c7fd27f75dbba586a31e769e1762780efec8aecafbfd59cd not found: ID does not exist"
Sep 09 21:08:17.291284 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:17.291261    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz"]
Sep 09 21:08:17.297221 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:17.297202    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-35e9ca-7c9c5765f8-c4mjz"]
Sep 09 21:08:18.892237 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:18.892206    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae" path="/var/lib/kubelet/pods/0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae/volumes"
Sep 09 21:08:18.892599 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:18.892537    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2ac943b-f4b4-4ba6-ad24-627a192232b0" path="/var/lib/kubelet/pods/d2ac943b-f4b4-4ba6-ad24-627a192232b0/volumes"
Sep 09 21:08:27.191691 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:08:27.191654    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:09:28.041515 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:28.041477    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ff038-s9rrf"]
Sep 09 21:09:28.041967 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:28.041864    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2ac943b-f4b4-4ba6-ad24-627a192232b0" containerName="revoke-keys"
Sep 09 21:09:28.041967 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:28.041875    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2ac943b-f4b4-4ba6-ad24-627a192232b0" containerName="revoke-keys"
Sep 09 21:09:28.041967 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:28.041891    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07b69822-5774-42cd-89cd-9915ba4e121a" containerName="istio-proxy"
Sep 09 21:09:28.041967 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:28.041896    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="07b69822-5774-42cd-89cd-9915ba4e121a" containerName="istio-proxy"
Sep 09 21:09:28.041967 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:28.041922    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae" containerName="payload-pre-processing"
Sep 09 21:09:28.041967 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:28.041928    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae" containerName="payload-pre-processing"
Sep 09 21:09:28.042167 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:28.042001    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="07b69822-5774-42cd-89cd-9915ba4e121a" containerName="istio-proxy"
Sep 09 21:09:28.042167 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:28.042010    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b43fcc6-ca1a-4f5d-8139-60b47c3dc9ae" containerName="payload-pre-processing"
Sep 09 21:09:28.042167 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:28.042016    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2ac943b-f4b4-4ba6-ad24-627a192232b0" containerName="revoke-keys"
Sep 09 21:09:28.045014 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:28.044989    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ff038-s9rrf"
Sep 09 21:09:28.047582 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:28.047557    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsks9\""
Sep 09 21:09:28.049025 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:28.049008    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 21:09:28.057631 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:28.057608    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ff038-s9rrf"]
Sep 09 21:09:28.114138 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:28.114110    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6e3274-c9d3-4f0a-a1a6-45e690ce3654-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2ff038-s9rrf\" (UID: \"5e6e3274-c9d3-4f0a-a1a6-45e690ce3654\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ff038-s9rrf"
Sep 09 21:09:28.215627 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:28.215605    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6e3274-c9d3-4f0a-a1a6-45e690ce3654-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2ff038-s9rrf\" (UID: \"5e6e3274-c9d3-4f0a-a1a6-45e690ce3654\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ff038-s9rrf"
Sep 09 21:09:28.216265 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:28.216244    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6e3274-c9d3-4f0a-a1a6-45e690ce3654-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2ff038-s9rrf\" (UID: \"5e6e3274-c9d3-4f0a-a1a6-45e690ce3654\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ff038-s9rrf"
Sep 09 21:09:28.355686 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:28.355616    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ff038-s9rrf"
Sep 09 21:09:28.631059 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:28.631033    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ff038-s9rrf"]
Sep 09 21:09:28.633603 ip-10-0-135-117 kubenswrapper[2556]: W0909 21:09:28.633576    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6e3274_c9d3_4f0a_a1a6_45e690ce3654.slice/crio-ec5c8f70309d0ea9e58ad3137db1ff1623a467d4182a1987da79ffc6af56c333 WatchSource:0}: Error finding container ec5c8f70309d0ea9e58ad3137db1ff1623a467d4182a1987da79ffc6af56c333: Status 404 returned error can't find the container with id ec5c8f70309d0ea9e58ad3137db1ff1623a467d4182a1987da79ffc6af56c333
Sep 09 21:09:28.642960 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:28.642933    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ff038-s9rrf" event={"ID":"5e6e3274-c9d3-4f0a-a1a6-45e690ce3654","Type":"ContainerStarted","Data":"ec5c8f70309d0ea9e58ad3137db1ff1623a467d4182a1987da79ffc6af56c333"}
Sep 09 21:09:29.649764 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:29.649675    2556 generic.go:356] "Generic (PLEG): container finished" podID="5e6e3274-c9d3-4f0a-a1a6-45e690ce3654" containerID="3a84a627c709fb91132e8ab9f0470e9811dd08e44460c5e5851100d0fb5f9b97" exitCode=0
Sep 09 21:09:29.650135 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:29.649758    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ff038-s9rrf" event={"ID":"5e6e3274-c9d3-4f0a-a1a6-45e690ce3654","Type":"ContainerDied","Data":"3a84a627c709fb91132e8ab9f0470e9811dd08e44460c5e5851100d0fb5f9b97"}
Sep 09 21:09:30.787281 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:30.787255    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ff038-s9rrf"
Sep 09 21:09:30.839002 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:30.838974    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6e3274-c9d3-4f0a-a1a6-45e690ce3654-maas-api-service-ca\") pod \"5e6e3274-c9d3-4f0a-a1a6-45e690ce3654\" (UID: \"5e6e3274-c9d3-4f0a-a1a6-45e690ce3654\") "
Sep 09 21:09:30.839281 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:30.839256    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6e3274-c9d3-4f0a-a1a6-45e690ce3654-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5e6e3274-c9d3-4f0a-a1a6-45e690ce3654" (UID: "5e6e3274-c9d3-4f0a-a1a6-45e690ce3654"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:09:30.939965 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:30.939895    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6e3274-c9d3-4f0a-a1a6-45e690ce3654-maas-api-service-ca\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:09:31.662520 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:31.662483    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ff038-s9rrf" event={"ID":"5e6e3274-c9d3-4f0a-a1a6-45e690ce3654","Type":"ContainerDied","Data":"ec5c8f70309d0ea9e58ad3137db1ff1623a467d4182a1987da79ffc6af56c333"}
Sep 09 21:09:31.662520 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:31.662518    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ff038-s9rrf"
Sep 09 21:09:31.662736 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:31.662543    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec5c8f70309d0ea9e58ad3137db1ff1623a467d4182a1987da79ffc6af56c333"
Sep 09 21:09:35.235061 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:35.235029    2556 scope.go:117] "RemoveContainer" containerID="0493b48338e47d0c2dfde51bada13ac485eb45b8ca46883e1af475d5103e07df"
Sep 09 21:09:38.786119 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:38.786084    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb"]
Sep 09 21:09:38.786589 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:38.786326    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb" podUID="76224359-80b5-49b1-93d0-855d0b45ff9e" containerName="payload-processing" containerID="cri-o://dbf2b107c38cbcca35b673bc1876d58feea75e4d1f795594572403efd18dbf50" gracePeriod=30
Sep 09 21:09:38.988580 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:38.988550    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ff038-s9rrf"]
Sep 09 21:09:38.994808 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:38.994784    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ff038-s9rrf"]
Sep 09 21:09:39.045630 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:39.045574    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb"
Sep 09 21:09:39.102220 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:39.102190    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76224359-80b5-49b1-93d0-855d0b45ff9e-plugins-config-volume\") pod \"76224359-80b5-49b1-93d0-855d0b45ff9e\" (UID: \"76224359-80b5-49b1-93d0-855d0b45ff9e\") "
Sep 09 21:09:39.102358 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:39.102342    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvdfk\" (UniqueName: \"kubernetes.io/projected/76224359-80b5-49b1-93d0-855d0b45ff9e-kube-api-access-pvdfk\") pod \"76224359-80b5-49b1-93d0-855d0b45ff9e\" (UID: \"76224359-80b5-49b1-93d0-855d0b45ff9e\") "
Sep 09 21:09:39.102570 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:39.102545    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76224359-80b5-49b1-93d0-855d0b45ff9e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "76224359-80b5-49b1-93d0-855d0b45ff9e" (UID: "76224359-80b5-49b1-93d0-855d0b45ff9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:09:39.102712 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:39.102692    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76224359-80b5-49b1-93d0-855d0b45ff9e-plugins-config-volume\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:09:39.104340 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:39.104323    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76224359-80b5-49b1-93d0-855d0b45ff9e-kube-api-access-pvdfk" (OuterVolumeSpecName: "kube-api-access-pvdfk") pod "76224359-80b5-49b1-93d0-855d0b45ff9e" (UID: "76224359-80b5-49b1-93d0-855d0b45ff9e"). InnerVolumeSpecName "kube-api-access-pvdfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:39.203830 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:39.203799    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvdfk\" (UniqueName: \"kubernetes.io/projected/76224359-80b5-49b1-93d0-855d0b45ff9e-kube-api-access-pvdfk\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:09:39.714371 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:39.714331    2556 generic.go:356] "Generic (PLEG): container finished" podID="76224359-80b5-49b1-93d0-855d0b45ff9e" containerID="dbf2b107c38cbcca35b673bc1876d58feea75e4d1f795594572403efd18dbf50" exitCode=0
Sep 09 21:09:39.714608 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:39.714572    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb" event={"ID":"76224359-80b5-49b1-93d0-855d0b45ff9e","Type":"ContainerDied","Data":"dbf2b107c38cbcca35b673bc1876d58feea75e4d1f795594572403efd18dbf50"}
Sep 09 21:09:39.714763 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:39.714619    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb"
Sep 09 21:09:39.714763 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:39.714714    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb" event={"ID":"76224359-80b5-49b1-93d0-855d0b45ff9e","Type":"ContainerDied","Data":"03b95bb131ae235f04b6c748363d601e954bac3b519e0a820e1207ebfadc25b1"}
Sep 09 21:09:39.714894 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:39.714769    2556 scope.go:117] "RemoveContainer" containerID="dbf2b107c38cbcca35b673bc1876d58feea75e4d1f795594572403efd18dbf50"
Sep 09 21:09:39.724547 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:39.724522    2556 scope.go:117] "RemoveContainer" containerID="dbf2b107c38cbcca35b673bc1876d58feea75e4d1f795594572403efd18dbf50"
Sep 09 21:09:39.724838 ip-10-0-135-117 kubenswrapper[2556]: E0909 21:09:39.724817    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbf2b107c38cbcca35b673bc1876d58feea75e4d1f795594572403efd18dbf50\": container with ID starting with dbf2b107c38cbcca35b673bc1876d58feea75e4d1f795594572403efd18dbf50 not found: ID does not exist" containerID="dbf2b107c38cbcca35b673bc1876d58feea75e4d1f795594572403efd18dbf50"
Sep 09 21:09:39.724929 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:39.724851    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf2b107c38cbcca35b673bc1876d58feea75e4d1f795594572403efd18dbf50"} err="failed to get container status \"dbf2b107c38cbcca35b673bc1876d58feea75e4d1f795594572403efd18dbf50\": rpc error: code = NotFound desc = could not find container \"dbf2b107c38cbcca35b673bc1876d58feea75e4d1f795594572403efd18dbf50\": container with ID starting with dbf2b107c38cbcca35b673bc1876d58feea75e4d1f795594572403efd18dbf50 not found: ID does not exist"
Sep 09 21:09:39.738601 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:39.738576    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb"]
Sep 09 21:09:39.742637 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:39.742617    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2ff038-c79cc6b77-ngjrb"]
Sep 09 21:09:40.892750 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:40.892705    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e6e3274-c9d3-4f0a-a1a6-45e690ce3654" path="/var/lib/kubelet/pods/5e6e3274-c9d3-4f0a-a1a6-45e690ce3654/volumes"
Sep 09 21:09:40.893302 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:40.893277    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76224359-80b5-49b1-93d0-855d0b45ff9e" path="/var/lib/kubelet/pods/76224359-80b5-49b1-93d0-855d0b45ff9e/volumes"
Sep 09 21:09:42.191197 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:42.191159    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:09:45.948902 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:45.948865    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cb82f2-pvfqs"]
Sep 09 21:09:45.949287 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:45.949251    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76224359-80b5-49b1-93d0-855d0b45ff9e" containerName="payload-processing"
Sep 09 21:09:45.949287 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:45.949263    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="76224359-80b5-49b1-93d0-855d0b45ff9e" containerName="payload-processing"
Sep 09 21:09:45.949287 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:45.949285    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07b69822-5774-42cd-89cd-9915ba4e121a" containerName="istio-proxy"
Sep 09 21:09:45.949398 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:45.949291    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="07b69822-5774-42cd-89cd-9915ba4e121a" containerName="istio-proxy"
Sep 09 21:09:45.949398 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:45.949306    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e6e3274-c9d3-4f0a-a1a6-45e690ce3654" containerName="revoke-keys"
Sep 09 21:09:45.949398 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:45.949314    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e6e3274-c9d3-4f0a-a1a6-45e690ce3654" containerName="revoke-keys"
Sep 09 21:09:45.949398 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:45.949366    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e6e3274-c9d3-4f0a-a1a6-45e690ce3654" containerName="revoke-keys"
Sep 09 21:09:45.949398 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:45.949375    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="07b69822-5774-42cd-89cd-9915ba4e121a" containerName="istio-proxy"
Sep 09 21:09:45.949398 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:45.949385    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="76224359-80b5-49b1-93d0-855d0b45ff9e" containerName="payload-processing"
Sep 09 21:09:45.952656 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:45.952638    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cb82f2-pvfqs"
Sep 09 21:09:45.955461 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:45.955439    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 21:09:45.955603 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:45.955441    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsks9\""
Sep 09 21:09:45.964922 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:45.963931    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cb82f2-pvfqs"]
Sep 09 21:09:46.066491 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:46.066452    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/231c1c84-a5ca-4ca8-a1a6-304169837521-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cb82f2-pvfqs\" (UID: \"231c1c84-a5ca-4ca8-a1a6-304169837521\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cb82f2-pvfqs"
Sep 09 21:09:46.167466 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:46.167426    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/231c1c84-a5ca-4ca8-a1a6-304169837521-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cb82f2-pvfqs\" (UID: \"231c1c84-a5ca-4ca8-a1a6-304169837521\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cb82f2-pvfqs"
Sep 09 21:09:46.168061 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:46.168042    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/231c1c84-a5ca-4ca8-a1a6-304169837521-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cb82f2-pvfqs\" (UID: \"231c1c84-a5ca-4ca8-a1a6-304169837521\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cb82f2-pvfqs"
Sep 09 21:09:46.269008 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:46.268981    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cb82f2-pvfqs"
Sep 09 21:09:46.539578 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:46.539550    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cb82f2-pvfqs"]
Sep 09 21:09:46.541613 ip-10-0-135-117 kubenswrapper[2556]: W0909 21:09:46.541582    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231c1c84_a5ca_4ca8_a1a6_304169837521.slice/crio-0b1fc955f9b5cc51a91cab948263a7152279b391d266bcdc058bd963199d459e WatchSource:0}: Error finding container 0b1fc955f9b5cc51a91cab948263a7152279b391d266bcdc058bd963199d459e: Status 404 returned error can't find the container with id 0b1fc955f9b5cc51a91cab948263a7152279b391d266bcdc058bd963199d459e
Sep 09 21:09:46.752784 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:46.752755    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cb82f2-pvfqs" event={"ID":"231c1c84-a5ca-4ca8-a1a6-304169837521","Type":"ContainerStarted","Data":"0b1fc955f9b5cc51a91cab948263a7152279b391d266bcdc058bd963199d459e"}
Sep 09 21:09:47.759718 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:47.759679    2556 generic.go:356] "Generic (PLEG): container finished" podID="231c1c84-a5ca-4ca8-a1a6-304169837521" containerID="46804a8857ab054f120296837ccc3a703485d464abf611f2f828bf72916685f2" exitCode=0
Sep 09 21:09:47.760106 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:47.759765    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cb82f2-pvfqs" event={"ID":"231c1c84-a5ca-4ca8-a1a6-304169837521","Type":"ContainerDied","Data":"46804a8857ab054f120296837ccc3a703485d464abf611f2f828bf72916685f2"}
Sep 09 21:09:48.891734 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:48.891710    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cb82f2-pvfqs"
Sep 09 21:09:48.995858 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:48.995830    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/231c1c84-a5ca-4ca8-a1a6-304169837521-maas-api-service-ca\") pod \"231c1c84-a5ca-4ca8-a1a6-304169837521\" (UID: \"231c1c84-a5ca-4ca8-a1a6-304169837521\") "
Sep 09 21:09:48.996168 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:48.996146    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/231c1c84-a5ca-4ca8-a1a6-304169837521-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "231c1c84-a5ca-4ca8-a1a6-304169837521" (UID: "231c1c84-a5ca-4ca8-a1a6-304169837521"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:09:49.096609 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:49.096551    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/231c1c84-a5ca-4ca8-a1a6-304169837521-maas-api-service-ca\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:09:49.771620 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:49.771584    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cb82f2-pvfqs" event={"ID":"231c1c84-a5ca-4ca8-a1a6-304169837521","Type":"ContainerDied","Data":"0b1fc955f9b5cc51a91cab948263a7152279b391d266bcdc058bd963199d459e"}
Sep 09 21:09:49.771620 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:49.771617    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cb82f2-pvfqs"
Sep 09 21:09:49.771810 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:49.771642    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b1fc955f9b5cc51a91cab948263a7152279b391d266bcdc058bd963199d459e"
Sep 09 21:09:56.795205 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:56.795170    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cb82f2-29816460-m4knn"]
Sep 09 21:09:56.797423 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:56.797394    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cb82f2-29816460-m4knn"]
Sep 09 21:09:56.822791 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:56.822762    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj"]
Sep 09 21:09:56.823073 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:56.823013    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj" podUID="2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3" containerName="payload-processing" containerID="cri-o://d7acb69071af33511e31df7c561e5f8d514a5e658965aab1a77758ae28eb976c" gracePeriod=30
Sep 09 21:09:56.892156 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:56.892121    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="105b2c9d-1d2c-49a5-b97e-7bb11473a158" path="/var/lib/kubelet/pods/105b2c9d-1d2c-49a5-b97e-7bb11473a158/volumes"
Sep 09 21:09:57.058518 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:57.058483    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cb82f2-pvfqs"]
Sep 09 21:09:57.060299 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:57.060273    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cb82f2-pvfqs"]
Sep 09 21:09:57.077414 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:57.077395    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj"
Sep 09 21:09:57.169580 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:57.169554    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqzbz\" (UniqueName: \"kubernetes.io/projected/2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3-kube-api-access-rqzbz\") pod \"2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3\" (UID: \"2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3\") "
Sep 09 21:09:57.169726 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:57.169639    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3-plugins-config-volume\") pod \"2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3\" (UID: \"2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3\") "
Sep 09 21:09:57.170049 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:57.170024    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3" (UID: "2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:09:57.171617 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:57.171591    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3-kube-api-access-rqzbz" (OuterVolumeSpecName: "kube-api-access-rqzbz") pod "2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3" (UID: "2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3"). InnerVolumeSpecName "kube-api-access-rqzbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:57.270798 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:57.270745    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3-plugins-config-volume\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:09:57.270798 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:57.270766    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqzbz\" (UniqueName: \"kubernetes.io/projected/2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3-kube-api-access-rqzbz\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:09:57.813790 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:57.813754    2556 generic.go:356] "Generic (PLEG): container finished" podID="2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3" containerID="d7acb69071af33511e31df7c561e5f8d514a5e658965aab1a77758ae28eb976c" exitCode=0
Sep 09 21:09:57.814211 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:57.813802    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj"
Sep 09 21:09:57.814211 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:57.813843    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj" event={"ID":"2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3","Type":"ContainerDied","Data":"d7acb69071af33511e31df7c561e5f8d514a5e658965aab1a77758ae28eb976c"}
Sep 09 21:09:57.814211 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:57.813898    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj" event={"ID":"2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3","Type":"ContainerDied","Data":"3aaf7a53d740ec556a40d72183ed9b0b7af2b1891c702555e233fbe1d4c5b253"}
Sep 09 21:09:57.814211 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:57.813931    2556 scope.go:117] "RemoveContainer" containerID="d7acb69071af33511e31df7c561e5f8d514a5e658965aab1a77758ae28eb976c"
Sep 09 21:09:57.823174 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:57.823150    2556 scope.go:117] "RemoveContainer" containerID="d7acb69071af33511e31df7c561e5f8d514a5e658965aab1a77758ae28eb976c"
Sep 09 21:09:57.829805 ip-10-0-135-117 kubenswrapper[2556]: E0909 21:09:57.829755    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7acb69071af33511e31df7c561e5f8d514a5e658965aab1a77758ae28eb976c\": container with ID starting with d7acb69071af33511e31df7c561e5f8d514a5e658965aab1a77758ae28eb976c not found: ID does not exist" containerID="d7acb69071af33511e31df7c561e5f8d514a5e658965aab1a77758ae28eb976c"
Sep 09 21:09:57.829805 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:57.829787    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7acb69071af33511e31df7c561e5f8d514a5e658965aab1a77758ae28eb976c"} err="failed to get container status \"d7acb69071af33511e31df7c561e5f8d514a5e658965aab1a77758ae28eb976c\": rpc error: code = NotFound desc = could not find container \"d7acb69071af33511e31df7c561e5f8d514a5e658965aab1a77758ae28eb976c\": container with ID starting with d7acb69071af33511e31df7c561e5f8d514a5e658965aab1a77758ae28eb976c not found: ID does not exist"
Sep 09 21:09:57.836888 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:57.836851    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj"]
Sep 09 21:09:57.843029 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:57.843007    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cb82f2-7ddfb7d6d5-blmcj"]
Sep 09 21:09:58.891502 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:58.891468    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="231c1c84-a5ca-4ca8-a1a6-304169837521" path="/var/lib/kubelet/pods/231c1c84-a5ca-4ca8-a1a6-304169837521/volumes"
Sep 09 21:09:58.891958 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:58.891812    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3" path="/var/lib/kubelet/pods/2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3/volumes"
Sep 09 21:09:59.789922 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:09:59.789883    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:10:03.937330 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:03.937297    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"]
Sep 09 21:10:03.937720 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:03.937498    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc" podUID="324975d3-3abe-433d-8f56-f75672e340ce" containerName="istio-proxy" containerID="cri-o://88d8523a1ee40d78d261f2cd7e2cbef6be0c637a3f886ee25d38a2a21f3389e5" gracePeriod=30
Sep 09 21:10:04.267486 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:04.267451    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a2fb5a-8j88k"]
Sep 09 21:10:04.267840 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:04.267827    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="231c1c84-a5ca-4ca8-a1a6-304169837521" containerName="revoke-keys"
Sep 09 21:10:04.267885 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:04.267844    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="231c1c84-a5ca-4ca8-a1a6-304169837521" containerName="revoke-keys"
Sep 09 21:10:04.267885 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:04.267870    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3" containerName="payload-processing"
Sep 09 21:10:04.267885 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:04.267876    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3" containerName="payload-processing"
Sep 09 21:10:04.267998 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:04.267949    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="231c1c84-a5ca-4ca8-a1a6-304169837521" containerName="revoke-keys"
Sep 09 21:10:04.267998 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:04.267961    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d6d8aeb-fbdd-47c5-87a5-fbcff33ec7e3" containerName="payload-processing"
Sep 09 21:10:04.272561 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:04.272528    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a2fb5a-8j88k"
Sep 09 21:10:04.275245 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:04.275219    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 21:10:04.275368 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:04.275297    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsks9\""
Sep 09 21:10:04.283774 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:04.282838    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a2fb5a-8j88k"]
Sep 09 21:10:04.332978 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:04.332953    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2879cf-176f-44f3-af3c-c7f15e758cc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a2fb5a-8j88k\" (UID: \"1c2879cf-176f-44f3-af3c-c7f15e758cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a2fb5a-8j88k"
Sep 09 21:10:04.433656 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:04.433631    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2879cf-176f-44f3-af3c-c7f15e758cc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a2fb5a-8j88k\" (UID: \"1c2879cf-176f-44f3-af3c-c7f15e758cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a2fb5a-8j88k"
Sep 09 21:10:04.434210 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:04.434193    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2879cf-176f-44f3-af3c-c7f15e758cc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a2fb5a-8j88k\" (UID: \"1c2879cf-176f-44f3-af3c-c7f15e758cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a2fb5a-8j88k"
Sep 09 21:10:04.590789 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:04.590706    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a2fb5a-8j88k"
Sep 09 21:10:04.864653 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:04.864629    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a2fb5a-8j88k"]
Sep 09 21:10:04.866939 ip-10-0-135-117 kubenswrapper[2556]: W0909 21:10:04.866893    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2879cf_176f_44f3_af3c_c7f15e758cc4.slice/crio-50d5e68638109114f7bdc9ba518b77a5b9c9ee2b03d2e4656a5d62fb5425046e WatchSource:0}: Error finding container 50d5e68638109114f7bdc9ba518b77a5b9c9ee2b03d2e4656a5d62fb5425046e: Status 404 returned error can't find the container with id 50d5e68638109114f7bdc9ba518b77a5b9c9ee2b03d2e4656a5d62fb5425046e
Sep 09 21:10:05.858558 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:05.858477    2556 generic.go:356] "Generic (PLEG): container finished" podID="1c2879cf-176f-44f3-af3c-c7f15e758cc4" containerID="816230f0a1c329d41fc7056ebce756b3a5c5ad2c9b616e5b72cfd7e2d98d52d5" exitCode=0
Sep 09 21:10:05.858934 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:05.858570    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a2fb5a-8j88k" event={"ID":"1c2879cf-176f-44f3-af3c-c7f15e758cc4","Type":"ContainerDied","Data":"816230f0a1c329d41fc7056ebce756b3a5c5ad2c9b616e5b72cfd7e2d98d52d5"}
Sep 09 21:10:05.858934 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:05.858631    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a2fb5a-8j88k" event={"ID":"1c2879cf-176f-44f3-af3c-c7f15e758cc4","Type":"ContainerStarted","Data":"50d5e68638109114f7bdc9ba518b77a5b9c9ee2b03d2e4656a5d62fb5425046e"}
Sep 09 21:10:06.997375 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:06.997352    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a2fb5a-8j88k"
Sep 09 21:10:07.057402 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:07.057376    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2879cf-176f-44f3-af3c-c7f15e758cc4-maas-api-service-ca\") pod \"1c2879cf-176f-44f3-af3c-c7f15e758cc4\" (UID: \"1c2879cf-176f-44f3-af3c-c7f15e758cc4\") "
Sep 09 21:10:07.057712 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:07.057692    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c2879cf-176f-44f3-af3c-c7f15e758cc4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1c2879cf-176f-44f3-af3c-c7f15e758cc4" (UID: "1c2879cf-176f-44f3-af3c-c7f15e758cc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:10:07.158953 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:07.158877    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2879cf-176f-44f3-af3c-c7f15e758cc4-maas-api-service-ca\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:10:07.871061 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:07.871029    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a2fb5a-8j88k" event={"ID":"1c2879cf-176f-44f3-af3c-c7f15e758cc4","Type":"ContainerDied","Data":"50d5e68638109114f7bdc9ba518b77a5b9c9ee2b03d2e4656a5d62fb5425046e"}
Sep 09 21:10:07.871234 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:07.871079    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50d5e68638109114f7bdc9ba518b77a5b9c9ee2b03d2e4656a5d62fb5425046e"
Sep 09 21:10:07.871234 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:07.871035    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a2fb5a-8j88k"
Sep 09 21:10:09.182485 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.182461    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 21:10:09.277034 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.276992    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-credential-socket\") pod \"324975d3-3abe-433d-8f56-f75672e340ce\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") "
Sep 09 21:10:09.277206 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.277075    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324975d3-3abe-433d-8f56-f75672e340ce-istio-podinfo\") pod \"324975d3-3abe-433d-8f56-f75672e340ce\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") "
Sep 09 21:10:09.277206 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.277108    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324975d3-3abe-433d-8f56-f75672e340ce-istiod-ca-cert\") pod \"324975d3-3abe-433d-8f56-f75672e340ce\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") "
Sep 09 21:10:09.277206 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.277151    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-workload-certs\") pod \"324975d3-3abe-433d-8f56-f75672e340ce\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") "
Sep 09 21:10:09.277206 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.277200    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-istio-data\") pod \"324975d3-3abe-433d-8f56-f75672e340ce\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") "
Sep 09 21:10:09.277428 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.277240    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "324975d3-3abe-433d-8f56-f75672e340ce" (UID: "324975d3-3abe-433d-8f56-f75672e340ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:09.277428 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.277249    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-istio-envoy\") pod \"324975d3-3abe-433d-8f56-f75672e340ce\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") "
Sep 09 21:10:09.277428 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.277337    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz8pd\" (UniqueName: \"kubernetes.io/projected/324975d3-3abe-433d-8f56-f75672e340ce-kube-api-access-dz8pd\") pod \"324975d3-3abe-433d-8f56-f75672e340ce\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") "
Sep 09 21:10:09.277428 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.277379    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324975d3-3abe-433d-8f56-f75672e340ce-istio-token\") pod \"324975d3-3abe-433d-8f56-f75672e340ce\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") "
Sep 09 21:10:09.277626 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.277424    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "324975d3-3abe-433d-8f56-f75672e340ce" (UID: "324975d3-3abe-433d-8f56-f75672e340ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:09.277626 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.277433    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-workload-socket\") pod \"324975d3-3abe-433d-8f56-f75672e340ce\" (UID: \"324975d3-3abe-433d-8f56-f75672e340ce\") "
Sep 09 21:10:09.277626 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.277466    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-istio-data" (OuterVolumeSpecName: "istio-data") pod "324975d3-3abe-433d-8f56-f75672e340ce" (UID: "324975d3-3abe-433d-8f56-f75672e340ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:09.277626 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.277487    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324975d3-3abe-433d-8f56-f75672e340ce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "324975d3-3abe-433d-8f56-f75672e340ce" (UID: "324975d3-3abe-433d-8f56-f75672e340ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:10:09.277831 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.277738    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324975d3-3abe-433d-8f56-f75672e340ce-istiod-ca-cert\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:10:09.277831 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.277740    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "324975d3-3abe-433d-8f56-f75672e340ce" (UID: "324975d3-3abe-433d-8f56-f75672e340ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:09.277831 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.277757    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-workload-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:10:09.277831 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.277769    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-istio-data\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:10:09.277831 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.277777    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-credential-socket\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:10:09.279227 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.279201    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/324975d3-3abe-433d-8f56-f75672e340ce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "324975d3-3abe-433d-8f56-f75672e340ce" (UID: "324975d3-3abe-433d-8f56-f75672e340ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:10:09.279676 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.279647    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324975d3-3abe-433d-8f56-f75672e340ce-istio-token" (OuterVolumeSpecName: "istio-token") pod "324975d3-3abe-433d-8f56-f75672e340ce" (UID: "324975d3-3abe-433d-8f56-f75672e340ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:10:09.279849 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.279826    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "324975d3-3abe-433d-8f56-f75672e340ce" (UID: "324975d3-3abe-433d-8f56-f75672e340ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:09.279849 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.279836    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324975d3-3abe-433d-8f56-f75672e340ce-kube-api-access-dz8pd" (OuterVolumeSpecName: "kube-api-access-dz8pd") pod "324975d3-3abe-433d-8f56-f75672e340ce" (UID: "324975d3-3abe-433d-8f56-f75672e340ce"). InnerVolumeSpecName "kube-api-access-dz8pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:10:09.379133 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.379098    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-workload-socket\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:10:09.379133 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.379129    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324975d3-3abe-433d-8f56-f75672e340ce-istio-podinfo\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:10:09.379133 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.379138    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324975d3-3abe-433d-8f56-f75672e340ce-istio-envoy\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:10:09.379309 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.379147    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz8pd\" (UniqueName: \"kubernetes.io/projected/324975d3-3abe-433d-8f56-f75672e340ce-kube-api-access-dz8pd\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:10:09.379309 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.379157    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324975d3-3abe-433d-8f56-f75672e340ce-istio-token\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:10:09.887466 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.887432    2556 generic.go:356] "Generic (PLEG): container finished" podID="324975d3-3abe-433d-8f56-f75672e340ce" containerID="88d8523a1ee40d78d261f2cd7e2cbef6be0c637a3f886ee25d38a2a21f3389e5" exitCode=0
Sep 09 21:10:09.887611 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.887493    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"
Sep 09 21:10:09.887611 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.887519    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc" event={"ID":"324975d3-3abe-433d-8f56-f75672e340ce","Type":"ContainerDied","Data":"88d8523a1ee40d78d261f2cd7e2cbef6be0c637a3f886ee25d38a2a21f3389e5"}
Sep 09 21:10:09.887611 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.887571    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc" event={"ID":"324975d3-3abe-433d-8f56-f75672e340ce","Type":"ContainerDied","Data":"6bc978f7299373ca304dbd52498024803d2024b5f44e652a1c15a9d68439cb37"}
Sep 09 21:10:09.887611 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.887600    2556 scope.go:117] "RemoveContainer" containerID="88d8523a1ee40d78d261f2cd7e2cbef6be0c637a3f886ee25d38a2a21f3389e5"
Sep 09 21:10:09.897566 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.897550    2556 scope.go:117] "RemoveContainer" containerID="d68894d83af93c791d7be1e46ed3d610f786bfa0d765c92c389a6c638c6306f0"
Sep 09 21:10:09.905594 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.905575    2556 scope.go:117] "RemoveContainer" containerID="88d8523a1ee40d78d261f2cd7e2cbef6be0c637a3f886ee25d38a2a21f3389e5"
Sep 09 21:10:09.905846 ip-10-0-135-117 kubenswrapper[2556]: E0909 21:10:09.905827    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d8523a1ee40d78d261f2cd7e2cbef6be0c637a3f886ee25d38a2a21f3389e5\": container with ID starting with 88d8523a1ee40d78d261f2cd7e2cbef6be0c637a3f886ee25d38a2a21f3389e5 not found: ID does not exist" containerID="88d8523a1ee40d78d261f2cd7e2cbef6be0c637a3f886ee25d38a2a21f3389e5"
Sep 09 21:10:09.905886 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.905856    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d8523a1ee40d78d261f2cd7e2cbef6be0c637a3f886ee25d38a2a21f3389e5"} err="failed to get container status \"88d8523a1ee40d78d261f2cd7e2cbef6be0c637a3f886ee25d38a2a21f3389e5\": rpc error: code = NotFound desc = could not find container \"88d8523a1ee40d78d261f2cd7e2cbef6be0c637a3f886ee25d38a2a21f3389e5\": container with ID starting with 88d8523a1ee40d78d261f2cd7e2cbef6be0c637a3f886ee25d38a2a21f3389e5 not found: ID does not exist"
Sep 09 21:10:09.905886 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.905872    2556 scope.go:117] "RemoveContainer" containerID="d68894d83af93c791d7be1e46ed3d610f786bfa0d765c92c389a6c638c6306f0"
Sep 09 21:10:09.906138 ip-10-0-135-117 kubenswrapper[2556]: E0909 21:10:09.906120    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d68894d83af93c791d7be1e46ed3d610f786bfa0d765c92c389a6c638c6306f0\": container with ID starting with d68894d83af93c791d7be1e46ed3d610f786bfa0d765c92c389a6c638c6306f0 not found: ID does not exist" containerID="d68894d83af93c791d7be1e46ed3d610f786bfa0d765c92c389a6c638c6306f0"
Sep 09 21:10:09.906182 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.906143    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d68894d83af93c791d7be1e46ed3d610f786bfa0d765c92c389a6c638c6306f0"} err="failed to get container status \"d68894d83af93c791d7be1e46ed3d610f786bfa0d765c92c389a6c638c6306f0\": rpc error: code = NotFound desc = could not find container \"d68894d83af93c791d7be1e46ed3d610f786bfa0d765c92c389a6c638c6306f0\": container with ID starting with d68894d83af93c791d7be1e46ed3d610f786bfa0d765c92c389a6c638c6306f0 not found: ID does not exist"
Sep 09 21:10:09.912469 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.912444    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"]
Sep 09 21:10:09.914247 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:09.914226    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cb82f2-openshift-default-fb89c9c4b-qmpvc"]
Sep 09 21:10:10.893759 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:10.893718    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="324975d3-3abe-433d-8f56-f75672e340ce" path="/var/lib/kubelet/pods/324975d3-3abe-433d-8f56-f75672e340ce/volumes"
Sep 09 21:10:15.098208 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:15.098174    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a2fb5a-8j88k"]
Sep 09 21:10:15.100672 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:15.100647    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a2fb5a-8j88k"]
Sep 09 21:10:16.893038 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:16.893004    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c2879cf-176f-44f3-af3c-c7f15e758cc4" path="/var/lib/kubelet/pods/1c2879cf-176f-44f3-af3c-c7f15e758cc4/volumes"
Sep 09 21:10:17.511533 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:17.511497    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:10:21.746544 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:21.746506    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"]
Sep 09 21:10:21.746941 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:21.746746    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql" podUID="5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" containerName="istio-proxy" containerID="cri-o://328380a79b50f1713af65f153b99c4ed7684579049c516a6ae2a85f0f669f61b" gracePeriod=30
Sep 09 21:10:26.410879 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:26.410839    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:10:26.978540 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:26.978508    2556 generic.go:356] "Generic (PLEG): container finished" podID="5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" containerID="328380a79b50f1713af65f153b99c4ed7684579049c516a6ae2a85f0f669f61b" exitCode=0
Sep 09 21:10:26.978670 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:26.978583    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql" event={"ID":"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b","Type":"ContainerDied","Data":"328380a79b50f1713af65f153b99c4ed7684579049c516a6ae2a85f0f669f61b"}
Sep 09 21:10:26.978717 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:26.978668    2556 scope.go:117] "RemoveContainer" containerID="dd36d59c46513f4fa5e8bc1a47935def791714dd0320a0fdbc855a095dbdcbcc"
Sep 09 21:10:26.998120 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:26.998100    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 21:10:27.132052 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.131982    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-workload-certs\") pod \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") "
Sep 09 21:10:27.132052 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.132024    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfnrl\" (UniqueName: \"kubernetes.io/projected/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-kube-api-access-rfnrl\") pod \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") "
Sep 09 21:10:27.132052 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.132052    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-workload-socket\") pod \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") "
Sep 09 21:10:27.132310 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.132072    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istiod-ca-cert\") pod \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") "
Sep 09 21:10:27.132310 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.132236    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-envoy\") pod \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") "
Sep 09 21:10:27.132421 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.132326    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-data\") pod \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") "
Sep 09 21:10:27.132421 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.132331    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" (UID: "5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:27.132421 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.132315    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" (UID: "5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:27.132421 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.132359    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-credential-socket\") pod \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") "
Sep 09 21:10:27.132421 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.132419    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-token\") pod \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") "
Sep 09 21:10:27.132688 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.132475    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-podinfo\") pod \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\" (UID: \"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b\") "
Sep 09 21:10:27.132688 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.132532    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-data" (OuterVolumeSpecName: "istio-data") pod "5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" (UID: "5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:27.132688 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.132560    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" (UID: "5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:27.132688 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.132576    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" (UID: "5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:10:27.132932 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.132724    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-credential-socket\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:10:27.132932 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.132736    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-workload-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:10:27.132932 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.132753    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-workload-socket\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:10:27.132932 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.132769    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istiod-ca-cert\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:10:27.132932 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.132785    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-data\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:10:27.134378 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.134352    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" (UID: "5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:27.134482 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.134400    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" (UID: "5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:10:27.134525 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.134492    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-token" (OuterVolumeSpecName: "istio-token") pod "5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" (UID: "5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:10:27.134560 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.134551    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-kube-api-access-rfnrl" (OuterVolumeSpecName: "kube-api-access-rfnrl") pod "5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" (UID: "5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b"). InnerVolumeSpecName "kube-api-access-rfnrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:10:27.233811 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.233788    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-token\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:10:27.233811 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.233809    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-podinfo\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:10:27.233967 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.233818    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfnrl\" (UniqueName: \"kubernetes.io/projected/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-kube-api-access-rfnrl\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:10:27.233967 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.233827    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b-istio-envoy\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:10:27.984948 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.984917    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"
Sep 09 21:10:27.984948 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.984931    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql" event={"ID":"5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b","Type":"ContainerDied","Data":"98d66a0d8b362b6b3c7747dfd578f3a7e28832072a46eaf81c7172f81eb0b0ab"}
Sep 09 21:10:27.985385 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:27.985007    2556 scope.go:117] "RemoveContainer" containerID="328380a79b50f1713af65f153b99c4ed7684579049c516a6ae2a85f0f669f61b"
Sep 09 21:10:28.021253 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:28.021224    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"]
Sep 09 21:10:28.026223 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:28.026202    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a2fb5a-openshift-default-7d6dffd476-xbdql"]
Sep 09 21:10:28.892406 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:28.892366    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" path="/var/lib/kubelet/pods/5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b/volumes"
Sep 09 21:10:35.258876 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:10:35.258846    2556 scope.go:117] "RemoveContainer" containerID="37b5fbcdab45fcf7555b30874971130b00104c38ebeb22b7e2de20677db6288f"
Sep 09 21:11:46.602421 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:11:46.602379    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:12:13.102442 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:12:13.102408    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:12:21.060297 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:12:21.060263    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:12:28.999645 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:12:28.999609    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:12:42.190489 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:12:42.190456    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:12:47.988720 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:12:47.988690    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:14:14.202992 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:14.202950    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:14:20.100224 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:20.100191    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:14:27.600076 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:27.599997    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:14:35.315023 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:35.314993    2556 scope.go:117] "RemoveContainer" containerID="a0124428d202cafc2a4f90be78f2a8c2193d029d17b183cb92c33c14880cd4db"
Sep 09 21:14:35.322730 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:35.322712    2556 scope.go:117] "RemoveContainer" containerID="2dc81f8952bc5ce52951bc3000d8faaef17810624429944f29903c8f4d53b06e"
Sep 09 21:14:52.396217 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.396182    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"]
Sep 09 21:14:52.396685 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.396573    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" containerName="istio-proxy"
Sep 09 21:14:52.396685 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.396586    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" containerName="istio-proxy"
Sep 09 21:14:52.396685 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.396597    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2879cf-176f-44f3-af3c-c7f15e758cc4" containerName="revoke-keys"
Sep 09 21:14:52.396685 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.396603    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2879cf-176f-44f3-af3c-c7f15e758cc4" containerName="revoke-keys"
Sep 09 21:14:52.396685 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.396611    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" containerName="istio-proxy"
Sep 09 21:14:52.396685 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.396618    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" containerName="istio-proxy"
Sep 09 21:14:52.396685 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.396637    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="324975d3-3abe-433d-8f56-f75672e340ce" containerName="istio-proxy"
Sep 09 21:14:52.396685 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.396642    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="324975d3-3abe-433d-8f56-f75672e340ce" containerName="istio-proxy"
Sep 09 21:14:52.396685 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.396655    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="324975d3-3abe-433d-8f56-f75672e340ce" containerName="istio-proxy"
Sep 09 21:14:52.396685 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.396661    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="324975d3-3abe-433d-8f56-f75672e340ce" containerName="istio-proxy"
Sep 09 21:14:52.397045 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.396722    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="324975d3-3abe-433d-8f56-f75672e340ce" containerName="istio-proxy"
Sep 09 21:14:52.397045 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.396731    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" containerName="istio-proxy"
Sep 09 21:14:52.397045 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.396742    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c2879cf-176f-44f3-af3c-c7f15e758cc4" containerName="revoke-keys"
Sep 09 21:14:52.400017 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.399985    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:14:52.405044 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.405021    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-91468989-kserve-self-signed-certs\""
Sep 09 21:14:52.416752 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.416727    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"]
Sep 09 21:14:52.471294 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.471265    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-tmp-dir\") pod \"e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") " pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:14:52.471397 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.471329    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj2dr\" (UniqueName: \"kubernetes.io/projected/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-kube-api-access-mj2dr\") pod \"e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") " pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:14:52.471397 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.471373    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-home\") pod \"e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") " pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:14:52.471397 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.471388    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-model-cache\") pod \"e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") " pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:14:52.471507 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.471415    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-dshm\") pod \"e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") " pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:14:52.471507 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.471431    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-tls-certs\") pod \"e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") " pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:14:52.572308 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.572276    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj2dr\" (UniqueName: \"kubernetes.io/projected/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-kube-api-access-mj2dr\") pod \"e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") " pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:14:52.572407 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.572321    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-home\") pod \"e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") " pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:14:52.572407 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.572344    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-model-cache\") pod \"e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") " pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:14:52.572407 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.572390    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-dshm\") pod \"e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") " pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:14:52.572548 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.572414    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-tls-certs\") pod \"e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") " pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:14:52.572548 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.572460    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-tmp-dir\") pod \"e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") " pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:14:52.572748 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.572724    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-home\") pod \"e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") " pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:14:52.572817 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.572750    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-model-cache\") pod \"e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") " pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:14:52.572875 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.572834    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-tmp-dir\") pod \"e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") " pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:14:52.574695 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.574670    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-dshm\") pod \"e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") " pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:14:52.574991 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.574970    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-tls-certs\") pod \"e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") " pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:14:52.583606 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.583587    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj2dr\" (UniqueName: \"kubernetes.io/projected/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-kube-api-access-mj2dr\") pod \"e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") " pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:14:52.709086 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.709032    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:14:52.978260 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:52.978201    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"]
Sep 09 21:14:52.980403 ip-10-0-135-117 kubenswrapper[2556]: W0909 21:14:52.980380    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3cdcf89b_2840_41f4_8d8f_55caa5841c2f.slice/crio-ddcfe347e194d365944da225debc60fb0709636ad4204214803c24ef0cc0fef5 WatchSource:0}: Error finding container ddcfe347e194d365944da225debc60fb0709636ad4204214803c24ef0cc0fef5: Status 404 returned error can't find the container with id ddcfe347e194d365944da225debc60fb0709636ad4204214803c24ef0cc0fef5
Sep 09 21:14:53.292528 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:53.292490    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf" event={"ID":"3cdcf89b-2840-41f4-8d8f-55caa5841c2f","Type":"ContainerStarted","Data":"a81e2f7103c925ea9490c93efafe0282cbda79436cd89a38aaf3ff01c6688c6e"}
Sep 09 21:14:53.292705 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:53.292538    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf" event={"ID":"3cdcf89b-2840-41f4-8d8f-55caa5841c2f","Type":"ContainerStarted","Data":"ddcfe347e194d365944da225debc60fb0709636ad4204214803c24ef0cc0fef5"}
Sep 09 21:14:53.313774 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:53.313717    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf" podStartSLOduration=1.3137029519999999 podStartE2EDuration="1.313702952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:14:53.310952566 +0000 UTC m=+2538.990912877" watchObservedRunningTime="2026-09-09 21:14:53.313702952 +0000 UTC m=+2538.993663263"
Sep 09 21:14:55.101062 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:14:55.101019    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:15:00.132496 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:00.132462    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816475-vl9zq"]
Sep 09 21:15:00.132967 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:00.132898    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="324975d3-3abe-433d-8f56-f75672e340ce" containerName="istio-proxy"
Sep 09 21:15:00.132967 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:00.132925    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ecb41f5-9336-41eb-8fdc-9a5a8c6d991b" containerName="istio-proxy"
Sep 09 21:15:00.136066 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:00.136048    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816475-vl9zq"
Sep 09 21:15:00.138893 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:00.138871    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsks9\""
Sep 09 21:15:00.142816 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:00.142793    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816475-vl9zq"]
Sep 09 21:15:00.148199 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:00.148178    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816475-vl9zq"
Sep 09 21:15:00.418637 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:00.418613    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816475-vl9zq"]
Sep 09 21:15:00.420706 ip-10-0-135-117 kubenswrapper[2556]: W0909 21:15:00.420676    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8746311a_7b87_42c1_9e71_43d5cf8b2f4d.slice/crio-5cd7ade8df6e7a9be24d8d18c9d8a5a68065242883c4a1a044b581e71ae0063e WatchSource:0}: Error finding container 5cd7ade8df6e7a9be24d8d18c9d8a5a68065242883c4a1a044b581e71ae0063e: Status 404 returned error can't find the container with id 5cd7ade8df6e7a9be24d8d18c9d8a5a68065242883c4a1a044b581e71ae0063e
Sep 09 21:15:01.332552 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:01.332522    2556 generic.go:356] "Generic (PLEG): container finished" podID="8746311a-7b87-42c1-9e71-43d5cf8b2f4d" containerID="65a25aaa3b99debb697fe2a9a0d5fc021ad65044480648a9b082699e610011d5" exitCode=0
Sep 09 21:15:01.332861 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:01.332562    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816475-vl9zq" event={"ID":"8746311a-7b87-42c1-9e71-43d5cf8b2f4d","Type":"ContainerDied","Data":"65a25aaa3b99debb697fe2a9a0d5fc021ad65044480648a9b082699e610011d5"}
Sep 09 21:15:01.332861 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:01.332600    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816475-vl9zq" event={"ID":"8746311a-7b87-42c1-9e71-43d5cf8b2f4d","Type":"ContainerStarted","Data":"5cd7ade8df6e7a9be24d8d18c9d8a5a68065242883c4a1a044b581e71ae0063e"}
Sep 09 21:15:02.462176 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:02.462153    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816475-vl9zq"
Sep 09 21:15:02.710203 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:02.710131    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:15:02.710203 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:02.710176    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:15:02.722736 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:02.722717    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:15:03.344016 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:03.343987    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816475-vl9zq"
Sep 09 21:15:03.344176 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:03.344028    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816475-vl9zq" event={"ID":"8746311a-7b87-42c1-9e71-43d5cf8b2f4d","Type":"ContainerDied","Data":"5cd7ade8df6e7a9be24d8d18c9d8a5a68065242883c4a1a044b581e71ae0063e"}
Sep 09 21:15:03.344176 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:03.344070    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cd7ade8df6e7a9be24d8d18c9d8a5a68065242883c4a1a044b581e71ae0063e"
Sep 09 21:15:03.355577 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:03.355552    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:15:14.668820 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:14.668787    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"]
Sep 09 21:15:16.594386 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:16.594354    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:15:23.795174 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:23.795133    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:15:29.689720 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:29.689664    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf" podUID="3cdcf89b-2840-41f4-8d8f-55caa5841c2f" containerName="main" containerID="cri-o://a81e2f7103c925ea9490c93efafe0282cbda79436cd89a38aaf3ff01c6688c6e" gracePeriod=45
Sep 09 21:15:29.938009 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:29.937985    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:15:30.086872 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.086842    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-model-cache\") pod \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") "
Sep 09 21:15:30.087061 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.086889    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-tmp-dir\") pod \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") "
Sep 09 21:15:30.087061 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.086932    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mj2dr\" (UniqueName: \"kubernetes.io/projected/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-kube-api-access-mj2dr\") pod \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") "
Sep 09 21:15:30.087061 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.086964    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-tls-certs\") pod \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") "
Sep 09 21:15:30.087061 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.086994    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-dshm\") pod \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") "
Sep 09 21:15:30.087061 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.087047    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-home\") pod \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\" (UID: \"3cdcf89b-2840-41f4-8d8f-55caa5841c2f\") "
Sep 09 21:15:30.087386 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.087177    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-model-cache" (OuterVolumeSpecName: "model-cache") pod "3cdcf89b-2840-41f4-8d8f-55caa5841c2f" (UID: "3cdcf89b-2840-41f4-8d8f-55caa5841c2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:30.087386 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.087190    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3cdcf89b-2840-41f4-8d8f-55caa5841c2f" (UID: "3cdcf89b-2840-41f4-8d8f-55caa5841c2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:30.087713 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.087398    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-model-cache\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:15:30.087713 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.087409    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-home" (OuterVolumeSpecName: "home") pod "3cdcf89b-2840-41f4-8d8f-55caa5841c2f" (UID: "3cdcf89b-2840-41f4-8d8f-55caa5841c2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:30.087713 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.087419    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-tmp-dir\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:15:30.089328 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.089303    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3cdcf89b-2840-41f4-8d8f-55caa5841c2f" (UID: "3cdcf89b-2840-41f4-8d8f-55caa5841c2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:15:30.089328 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.089314    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-kube-api-access-mj2dr" (OuterVolumeSpecName: "kube-api-access-mj2dr") pod "3cdcf89b-2840-41f4-8d8f-55caa5841c2f" (UID: "3cdcf89b-2840-41f4-8d8f-55caa5841c2f"). InnerVolumeSpecName "kube-api-access-mj2dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:15:30.089489 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.089355    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-dshm" (OuterVolumeSpecName: "dshm") pod "3cdcf89b-2840-41f4-8d8f-55caa5841c2f" (UID: "3cdcf89b-2840-41f4-8d8f-55caa5841c2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:30.188055 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.188032    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-home\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:15:30.188055 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.188053    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj2dr\" (UniqueName: \"kubernetes.io/projected/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-kube-api-access-mj2dr\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:15:30.188189 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.188063    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-tls-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:15:30.188189 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.188071    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cdcf89b-2840-41f4-8d8f-55caa5841c2f-dshm\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:15:30.481286 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.481210    2556 generic.go:356] "Generic (PLEG): container finished" podID="3cdcf89b-2840-41f4-8d8f-55caa5841c2f" containerID="a81e2f7103c925ea9490c93efafe0282cbda79436cd89a38aaf3ff01c6688c6e" exitCode=0
Sep 09 21:15:30.481286 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.481259    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"
Sep 09 21:15:30.481286 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.481274    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf" event={"ID":"3cdcf89b-2840-41f4-8d8f-55caa5841c2f","Type":"ContainerDied","Data":"a81e2f7103c925ea9490c93efafe0282cbda79436cd89a38aaf3ff01c6688c6e"}
Sep 09 21:15:30.481482 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.481323    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf" event={"ID":"3cdcf89b-2840-41f4-8d8f-55caa5841c2f","Type":"ContainerDied","Data":"ddcfe347e194d365944da225debc60fb0709636ad4204214803c24ef0cc0fef5"}
Sep 09 21:15:30.481482 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.481346    2556 scope.go:117] "RemoveContainer" containerID="a81e2f7103c925ea9490c93efafe0282cbda79436cd89a38aaf3ff01c6688c6e"
Sep 09 21:15:30.490281 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.490262    2556 scope.go:117] "RemoveContainer" containerID="a81e2f7103c925ea9490c93efafe0282cbda79436cd89a38aaf3ff01c6688c6e"
Sep 09 21:15:30.490558 ip-10-0-135-117 kubenswrapper[2556]: E0909 21:15:30.490538    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a81e2f7103c925ea9490c93efafe0282cbda79436cd89a38aaf3ff01c6688c6e\": container with ID starting with a81e2f7103c925ea9490c93efafe0282cbda79436cd89a38aaf3ff01c6688c6e not found: ID does not exist" containerID="a81e2f7103c925ea9490c93efafe0282cbda79436cd89a38aaf3ff01c6688c6e"
Sep 09 21:15:30.490642 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.490566    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a81e2f7103c925ea9490c93efafe0282cbda79436cd89a38aaf3ff01c6688c6e"} err="failed to get container status \"a81e2f7103c925ea9490c93efafe0282cbda79436cd89a38aaf3ff01c6688c6e\": rpc error: code = NotFound desc = could not find container \"a81e2f7103c925ea9490c93efafe0282cbda79436cd89a38aaf3ff01c6688c6e\": container with ID starting with a81e2f7103c925ea9490c93efafe0282cbda79436cd89a38aaf3ff01c6688c6e not found: ID does not exist"
Sep 09 21:15:30.502027 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.502006    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"]
Sep 09 21:15:30.506144 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.506121    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-91468989-kserve-8df8f866c-l6lpf"]
Sep 09 21:15:30.892017 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:30.891988    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cdcf89b-2840-41f4-8d8f-55caa5841c2f" path="/var/lib/kubelet/pods/3cdcf89b-2840-41f4-8d8f-55caa5841c2f/volumes"
Sep 09 21:15:31.404840 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.404807    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:15:31.696895 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.696823    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"]
Sep 09 21:15:31.697226 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.697212    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8746311a-7b87-42c1-9e71-43d5cf8b2f4d" containerName="cleanup"
Sep 09 21:15:31.697280 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.697227    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8746311a-7b87-42c1-9e71-43d5cf8b2f4d" containerName="cleanup"
Sep 09 21:15:31.697280 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.697260    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cdcf89b-2840-41f4-8d8f-55caa5841c2f" containerName="main"
Sep 09 21:15:31.697280 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.697265    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cdcf89b-2840-41f4-8d8f-55caa5841c2f" containerName="main"
Sep 09 21:15:31.697375 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.697335    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cdcf89b-2840-41f4-8d8f-55caa5841c2f" containerName="main"
Sep 09 21:15:31.697375 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.697344    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8746311a-7b87-42c1-9e71-43d5cf8b2f4d" containerName="cleanup"
Sep 09 21:15:31.701508 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.701490    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"
Sep 09 21:15:31.705420 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.705386    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 21:15:31.705420 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.705415    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 21:15:31.705874 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.705847    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 21:15:31.706017 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.705849    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pzpjp\""
Sep 09 21:15:31.706017 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.705893    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 21:15:31.713244 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.713221    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"]
Sep 09 21:15:31.802600 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.802571    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cggj6\" (UniqueName: \"kubernetes.io/projected/b8ceaba2-5616-4ae7-95fa-870944033023-kube-api-access-cggj6\") pod \"maas-api-6fc5cc7c7f-2dcbh\" (UID: \"b8ceaba2-5616-4ae7-95fa-870944033023\") " pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"
Sep 09 21:15:31.802753 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.802611    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ceaba2-5616-4ae7-95fa-870944033023-openshift-service-ca\") pod \"maas-api-6fc5cc7c7f-2dcbh\" (UID: \"b8ceaba2-5616-4ae7-95fa-870944033023\") " pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"
Sep 09 21:15:31.802753 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.802639    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ceaba2-5616-4ae7-95fa-870944033023-maas-api-tls\") pod \"maas-api-6fc5cc7c7f-2dcbh\" (UID: \"b8ceaba2-5616-4ae7-95fa-870944033023\") " pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"
Sep 09 21:15:31.802753 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.802697    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ceaba2-5616-4ae7-95fa-870944033023-metrics-certs\") pod \"maas-api-6fc5cc7c7f-2dcbh\" (UID: \"b8ceaba2-5616-4ae7-95fa-870944033023\") " pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"
Sep 09 21:15:31.903984 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.903957    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cggj6\" (UniqueName: \"kubernetes.io/projected/b8ceaba2-5616-4ae7-95fa-870944033023-kube-api-access-cggj6\") pod \"maas-api-6fc5cc7c7f-2dcbh\" (UID: \"b8ceaba2-5616-4ae7-95fa-870944033023\") " pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"
Sep 09 21:15:31.904377 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.903996    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ceaba2-5616-4ae7-95fa-870944033023-openshift-service-ca\") pod \"maas-api-6fc5cc7c7f-2dcbh\" (UID: \"b8ceaba2-5616-4ae7-95fa-870944033023\") " pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"
Sep 09 21:15:31.904377 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.904015    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ceaba2-5616-4ae7-95fa-870944033023-maas-api-tls\") pod \"maas-api-6fc5cc7c7f-2dcbh\" (UID: \"b8ceaba2-5616-4ae7-95fa-870944033023\") " pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"
Sep 09 21:15:31.904377 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.904034    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ceaba2-5616-4ae7-95fa-870944033023-metrics-certs\") pod \"maas-api-6fc5cc7c7f-2dcbh\" (UID: \"b8ceaba2-5616-4ae7-95fa-870944033023\") " pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"
Sep 09 21:15:31.904625 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.904600    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ceaba2-5616-4ae7-95fa-870944033023-openshift-service-ca\") pod \"maas-api-6fc5cc7c7f-2dcbh\" (UID: \"b8ceaba2-5616-4ae7-95fa-870944033023\") " pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"
Sep 09 21:15:31.906496 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.906468    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ceaba2-5616-4ae7-95fa-870944033023-metrics-certs\") pod \"maas-api-6fc5cc7c7f-2dcbh\" (UID: \"b8ceaba2-5616-4ae7-95fa-870944033023\") " pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"
Sep 09 21:15:31.906618 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.906589    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ceaba2-5616-4ae7-95fa-870944033023-maas-api-tls\") pod \"maas-api-6fc5cc7c7f-2dcbh\" (UID: \"b8ceaba2-5616-4ae7-95fa-870944033023\") " pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"
Sep 09 21:15:31.912837 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:31.912816    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cggj6\" (UniqueName: \"kubernetes.io/projected/b8ceaba2-5616-4ae7-95fa-870944033023-kube-api-access-cggj6\") pod \"maas-api-6fc5cc7c7f-2dcbh\" (UID: \"b8ceaba2-5616-4ae7-95fa-870944033023\") " pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"
Sep 09 21:15:32.012477 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:32.012452    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"
Sep 09 21:15:32.287860 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:32.287835    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"]
Sep 09 21:15:32.289993 ip-10-0-135-117 kubenswrapper[2556]: W0909 21:15:32.289966    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ceaba2_5616_4ae7_95fa_870944033023.slice/crio-af710dcd4235bbd971678ccbd447fc80a393659d93f2c0e333c5066d59ce7363 WatchSource:0}: Error finding container af710dcd4235bbd971678ccbd447fc80a393659d93f2c0e333c5066d59ce7363: Status 404 returned error can't find the container with id af710dcd4235bbd971678ccbd447fc80a393659d93f2c0e333c5066d59ce7363
Sep 09 21:15:32.291442 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:32.291421    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:15:32.492075 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:32.492041    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh" event={"ID":"b8ceaba2-5616-4ae7-95fa-870944033023","Type":"ContainerStarted","Data":"af710dcd4235bbd971678ccbd447fc80a393659d93f2c0e333c5066d59ce7363"}
Sep 09 21:15:33.498572 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:33.498534    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh" event={"ID":"b8ceaba2-5616-4ae7-95fa-870944033023","Type":"ContainerStarted","Data":"cb8cb10f6fd94eabfc75b89aab3c5cc11dee1389bdd64b88a2a5cd3b1e91a333"}
Sep 09 21:15:33.499052 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:33.498601    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"
Sep 09 21:15:33.521304 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:33.521253    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh" podStartSLOduration=1.928038462 podStartE2EDuration="2.521238747s" totalImagesPullingTime="593.200285ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:15:33.51832343 +0000 UTC m=+2579.198283738" watchObservedRunningTime="2026-09-09 21:15:33.521238747 +0000 UTC m=+2579.201199055"
Sep 09 21:15:35.344375 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:35.344343    2556 scope.go:117] "RemoveContainer" containerID="3a84a627c709fb91132e8ab9f0470e9811dd08e44460c5e5851100d0fb5f9b97"
Sep 09 21:15:39.100435 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:39.100397    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:15:39.508258 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:39.508231    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"
Sep 09 21:15:45.986658 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:45.986622    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:15:46.694156 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:46.694117    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"]
Sep 09 21:15:46.694370 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:46.694345    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh" podUID="b8ceaba2-5616-4ae7-95fa-870944033023" containerName="maas-api" containerID="cri-o://cb8cb10f6fd94eabfc75b89aab3c5cc11dee1389bdd64b88a2a5cd3b1e91a333" gracePeriod=30
Sep 09 21:15:46.939635 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:46.939612    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"
Sep 09 21:15:47.032950 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:47.032924    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ceaba2-5616-4ae7-95fa-870944033023-metrics-certs\") pod \"b8ceaba2-5616-4ae7-95fa-870944033023\" (UID: \"b8ceaba2-5616-4ae7-95fa-870944033023\") "
Sep 09 21:15:47.033364 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:47.032991    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ceaba2-5616-4ae7-95fa-870944033023-openshift-service-ca\") pod \"b8ceaba2-5616-4ae7-95fa-870944033023\" (UID: \"b8ceaba2-5616-4ae7-95fa-870944033023\") "
Sep 09 21:15:47.033364 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:47.033053    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ceaba2-5616-4ae7-95fa-870944033023-maas-api-tls\") pod \"b8ceaba2-5616-4ae7-95fa-870944033023\" (UID: \"b8ceaba2-5616-4ae7-95fa-870944033023\") "
Sep 09 21:15:47.033364 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:47.033324    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8ceaba2-5616-4ae7-95fa-870944033023-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b8ceaba2-5616-4ae7-95fa-870944033023" (UID: "b8ceaba2-5616-4ae7-95fa-870944033023"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:15:47.035041 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:47.035020    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ceaba2-5616-4ae7-95fa-870944033023-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b8ceaba2-5616-4ae7-95fa-870944033023" (UID: "b8ceaba2-5616-4ae7-95fa-870944033023"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:15:47.035203 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:47.035185    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ceaba2-5616-4ae7-95fa-870944033023-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b8ceaba2-5616-4ae7-95fa-870944033023" (UID: "b8ceaba2-5616-4ae7-95fa-870944033023"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:15:47.133600 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:47.133577    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cggj6\" (UniqueName: \"kubernetes.io/projected/b8ceaba2-5616-4ae7-95fa-870944033023-kube-api-access-cggj6\") pod \"b8ceaba2-5616-4ae7-95fa-870944033023\" (UID: \"b8ceaba2-5616-4ae7-95fa-870944033023\") "
Sep 09 21:15:47.133851 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:47.133833    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ceaba2-5616-4ae7-95fa-870944033023-openshift-service-ca\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:15:47.133925 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:47.133857    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ceaba2-5616-4ae7-95fa-870944033023-maas-api-tls\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:15:47.133925 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:47.133870    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ceaba2-5616-4ae7-95fa-870944033023-metrics-certs\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:15:47.135251 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:47.135231    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ceaba2-5616-4ae7-95fa-870944033023-kube-api-access-cggj6" (OuterVolumeSpecName: "kube-api-access-cggj6") pod "b8ceaba2-5616-4ae7-95fa-870944033023" (UID: "b8ceaba2-5616-4ae7-95fa-870944033023"). InnerVolumeSpecName "kube-api-access-cggj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:15:47.234454 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:47.234429    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cggj6\" (UniqueName: \"kubernetes.io/projected/b8ceaba2-5616-4ae7-95fa-870944033023-kube-api-access-cggj6\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\""
Sep 09 21:15:47.574170 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:47.574134    2556 generic.go:356] "Generic (PLEG): container finished" podID="b8ceaba2-5616-4ae7-95fa-870944033023" containerID="cb8cb10f6fd94eabfc75b89aab3c5cc11dee1389bdd64b88a2a5cd3b1e91a333" exitCode=0
Sep 09 21:15:47.574321 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:47.574184    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"
Sep 09 21:15:47.574321 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:47.574221    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh" event={"ID":"b8ceaba2-5616-4ae7-95fa-870944033023","Type":"ContainerDied","Data":"cb8cb10f6fd94eabfc75b89aab3c5cc11dee1389bdd64b88a2a5cd3b1e91a333"}
Sep 09 21:15:47.574321 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:47.574281    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh" event={"ID":"b8ceaba2-5616-4ae7-95fa-870944033023","Type":"ContainerDied","Data":"af710dcd4235bbd971678ccbd447fc80a393659d93f2c0e333c5066d59ce7363"}
Sep 09 21:15:47.574321 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:47.574306    2556 scope.go:117] "RemoveContainer" containerID="cb8cb10f6fd94eabfc75b89aab3c5cc11dee1389bdd64b88a2a5cd3b1e91a333"
Sep 09 21:15:47.583896 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:47.583880    2556 scope.go:117] "RemoveContainer" containerID="cb8cb10f6fd94eabfc75b89aab3c5cc11dee1389bdd64b88a2a5cd3b1e91a333"
Sep 09 21:15:47.584205 ip-10-0-135-117 kubenswrapper[2556]: E0909 21:15:47.584187    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb8cb10f6fd94eabfc75b89aab3c5cc11dee1389bdd64b88a2a5cd3b1e91a333\": container with ID starting with cb8cb10f6fd94eabfc75b89aab3c5cc11dee1389bdd64b88a2a5cd3b1e91a333 not found: ID does not exist" containerID="cb8cb10f6fd94eabfc75b89aab3c5cc11dee1389bdd64b88a2a5cd3b1e91a333"
Sep 09 21:15:47.584256 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:47.584213    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb8cb10f6fd94eabfc75b89aab3c5cc11dee1389bdd64b88a2a5cd3b1e91a333"} err="failed to get container status \"cb8cb10f6fd94eabfc75b89aab3c5cc11dee1389bdd64b88a2a5cd3b1e91a333\": rpc error: code = NotFound desc = could not find container \"cb8cb10f6fd94eabfc75b89aab3c5cc11dee1389bdd64b88a2a5cd3b1e91a333\": container with ID starting with cb8cb10f6fd94eabfc75b89aab3c5cc11dee1389bdd64b88a2a5cd3b1e91a333 not found: ID does not exist"
Sep 09 21:15:47.596493 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:47.596465    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"]
Sep 09 21:15:47.599478 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:47.599458    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fc5cc7c7f-2dcbh"]
Sep 09 21:15:48.892426 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:48.892397    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8ceaba2-5616-4ae7-95fa-870944033023" path="/var/lib/kubelet/pods/b8ceaba2-5616-4ae7-95fa-870944033023/volumes"
Sep 09 21:15:51.495193 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:51.495158    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:15:58.690893 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:15:58.690812    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tpptr"]
Sep 09 21:16:07.116248 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:07.116220    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-859cf57674-j278h_50d78a07-27e0-41e1-9ecf-a9b9882eda59/copy-manifests/0.log"
Sep 09 21:16:07.122355 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:07.122333    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-859cf57674-j278h_50d78a07-27e0-41e1-9ecf-a9b9882eda59/manager/0.log"
Sep 09 21:16:07.649550 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:07.649523    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29816445-jxdd9_4caea27e-6bc7-4f35-a4da-5dd650051fee/cleanup/0.log"
Sep 09 21:16:07.846847 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:07.846816    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29816475-vl9zq_8746311a-7b87-42c1-9e71-43d5cf8b2f4d/cleanup/0.log"
Sep 09 21:16:09.005953 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:09.005928    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-zn5ms_ce3f6259-dd9e-4305-9348-472835f0f076/kuadrant-console-plugin/0.log"
Sep 09 21:16:09.241967 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:09.241937    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-tpptr_d63afcf8-a51a-48cb-a24f-4a222cbf2600/limitador/0.log"
Sep 09 21:16:10.018501 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:10.018477    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65d657c8d5-ptlv5_b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d/kube-auth-proxy/0.log"
Sep 09 21:16:10.894169 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:10.894137    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-9kn2j_09f27f65-11bd-43aa-94fa-2a903b79be77/main/0.log"
Sep 09 21:16:10.998550 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:10.998526    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-8l5rz_f3db7f59-5eea-4681-9d55-da11e9093913/main/0.log"
Sep 09 21:16:11.424313 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:11.424286    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-hpmlc_9176bbb6-3d05-4262-afee-1ac4717ce6c2/main/0.log"
Sep 09 21:16:19.877349 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:19.877315    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-q2zw5_61bb5c63-cbb7-4186-9c23-5706dba8cec2/global-pull-secret-syncer/0.log"
Sep 09 21:16:20.042588 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:20.042560    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jr87x_7241694e-5bba-4400-9eb9-771af5989280/konnectivity-agent/0.log"
Sep 09 21:16:20.132203 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:20.132138    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-117.ec2.internal_4a08d7fc3d09b5f1580b5cd035f70a19/haproxy/0.log"
Sep 09 21:16:24.935723 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:24.935691    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-zn5ms_ce3f6259-dd9e-4305-9348-472835f0f076/kuadrant-console-plugin/0.log"
Sep 09 21:16:25.011367 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:25.011342    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-tpptr_d63afcf8-a51a-48cb-a24f-4a222cbf2600/limitador/0.log"
Sep 09 21:16:26.389679 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:26.389652    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7607422d-c966-40af-a7ff-3e3e854743c2/alertmanager/0.log"
Sep 09 21:16:26.418065 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:26.418046    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7607422d-c966-40af-a7ff-3e3e854743c2/config-reloader/0.log"
Sep 09 21:16:26.439401 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:26.439364    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7607422d-c966-40af-a7ff-3e3e854743c2/kube-rbac-proxy-web/0.log"
Sep 09 21:16:26.462438 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:26.462413    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7607422d-c966-40af-a7ff-3e3e854743c2/kube-rbac-proxy/0.log"
Sep 09 21:16:26.484080 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:26.484061    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7607422d-c966-40af-a7ff-3e3e854743c2/kube-rbac-proxy-metric/0.log"
Sep 09 21:16:26.506329 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:26.506310    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7607422d-c966-40af-a7ff-3e3e854743c2/prom-label-proxy/0.log"
Sep 09 21:16:26.532916 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:26.532886    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7607422d-c966-40af-a7ff-3e3e854743c2/init-config-reloader/0.log"
Sep 09 21:16:26.614312 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:26.614290    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-4fwvm_c5ebe346-482a-410a-80b8-7b0a2981a38c/kube-state-metrics/0.log"
Sep 09 21:16:26.641742 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:26.641684    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-4fwvm_c5ebe346-482a-410a-80b8-7b0a2981a38c/kube-rbac-proxy-main/0.log"
Sep 09 21:16:26.667012 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:26.666996    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-4fwvm_c5ebe346-482a-410a-80b8-7b0a2981a38c/kube-rbac-proxy-self/0.log"
Sep 09 21:16:26.698483 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:26.698463    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5d64f9bfdb-v2gn4_cbc0a8b7-eeab-44dc-9cbe-c3578953fd03/metrics-server/0.log"
Sep 09 21:16:26.733400 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:26.733380    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-tmchm_e7a00cf1-7c14-49d7-9868-adc97c234e3f/monitoring-plugin/0.log"
Sep 09 21:16:27.157970 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:27.157872    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cxbm6_21002114-9907-455c-9324-a5ccb05b9516/node-exporter/0.log"
Sep 09 21:16:27.219978 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:27.219953    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cxbm6_21002114-9907-455c-9324-a5ccb05b9516/kube-rbac-proxy/0.log"
Sep 09 21:16:27.269616 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:27.269597    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cxbm6_21002114-9907-455c-9324-a5ccb05b9516/init-textfile/0.log"
Sep 09 21:16:27.324444 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:27.324423    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-mv87t_f2a295c8-1860-4315-ab35-764265d474e9/kube-rbac-proxy-main/0.log"
Sep 09 21:16:27.385700 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:27.385676    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-mv87t_f2a295c8-1860-4315-ab35-764265d474e9/kube-rbac-proxy-self/0.log"
Sep 09 21:16:27.425429 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:27.425356    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-mv87t_f2a295c8-1860-4315-ab35-764265d474e9/openshift-state-metrics/0.log"
Sep 09 21:16:27.485746 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:27.485721    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_82880fb9-9943-4d5d-b23e-b5c3c05319fc/prometheus/0.log"
Sep 09 21:16:27.538824 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:27.538800    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_82880fb9-9943-4d5d-b23e-b5c3c05319fc/config-reloader/0.log"
Sep 09 21:16:27.601652 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:27.601629    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_82880fb9-9943-4d5d-b23e-b5c3c05319fc/thanos-sidecar/0.log"
Sep 09 21:16:27.636672 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:27.636650    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_82880fb9-9943-4d5d-b23e-b5c3c05319fc/kube-rbac-proxy-web/0.log"
Sep 09 21:16:27.675704 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:27.675648    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_82880fb9-9943-4d5d-b23e-b5c3c05319fc/kube-rbac-proxy/0.log"
Sep 09 21:16:27.714987 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:27.714963    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_82880fb9-9943-4d5d-b23e-b5c3c05319fc/kube-rbac-proxy-thanos/0.log"
Sep 09 21:16:27.754340 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:27.754320    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_82880fb9-9943-4d5d-b23e-b5c3c05319fc/init-config-reloader/0.log"
Sep 09 21:16:27.817500 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:27.817481    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-jlrwd_a4201a6a-7846-48fb-801f-4775039db982/prometheus-operator/0.log"
Sep 09 21:16:27.847040 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:27.847024    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-jlrwd_a4201a6a-7846-48fb-801f-4775039db982/kube-rbac-proxy/0.log"
Sep 09 21:16:27.875577 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:27.875557    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-f75md_c687fc88-83ce-40e9-92bf-4a9802d1e021/prometheus-operator-admission-webhook/0.log"
Sep 09 21:16:28.002625 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:28.002604    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55dddc5f75-fxg4r_fa67c526-c03d-42ce-8c62-ccd903c21c5f/thanos-query/0.log"
Sep 09 21:16:28.030511 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:28.030485    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55dddc5f75-fxg4r_fa67c526-c03d-42ce-8c62-ccd903c21c5f/kube-rbac-proxy-web/0.log"
Sep 09 21:16:28.062578 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:28.062557    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55dddc5f75-fxg4r_fa67c526-c03d-42ce-8c62-ccd903c21c5f/kube-rbac-proxy/0.log"
Sep 09 21:16:28.087876 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:28.087858    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55dddc5f75-fxg4r_fa67c526-c03d-42ce-8c62-ccd903c21c5f/prom-label-proxy/0.log"
Sep 09 21:16:28.114085 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:28.114069    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55dddc5f75-fxg4r_fa67c526-c03d-42ce-8c62-ccd903c21c5f/kube-rbac-proxy-rules/0.log"
Sep 09 21:16:28.141763 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:28.141740    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55dddc5f75-fxg4r_fa67c526-c03d-42ce-8c62-ccd903c21c5f/kube-rbac-proxy-metrics/0.log"
Sep 09 21:16:30.117293 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.117258    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz"]
Sep 09 21:16:30.117698 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.117682    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8ceaba2-5616-4ae7-95fa-870944033023" containerName="maas-api"
Sep 09 21:16:30.117745 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.117700    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8ceaba2-5616-4ae7-95fa-870944033023" containerName="maas-api"
Sep 09 21:16:30.117782 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.117759    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8ceaba2-5616-4ae7-95fa-870944033023" containerName="maas-api"
Sep 09 21:16:30.120991 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.120969    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz"
Sep 09 21:16:30.125199 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.125174    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8prvh\"/\"kube-root-ca.crt\""
Sep 09 21:16:30.125311 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.125213    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8prvh\"/\"openshift-service-ca.crt\""
Sep 09 21:16:30.125311 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.125223    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8prvh\"/\"default-dockercfg-lr2bl\""
Sep 09 21:16:30.134164 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.134140    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz"]
Sep 09 21:16:30.266022 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.265995    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ecad8020-1da4-4c41-85ff-1edbf776141b-proc\") pod \"perf-node-gather-daemonset-vpqkz\" (UID: \"ecad8020-1da4-4c41-85ff-1edbf776141b\") " pod="openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz"
Sep 09 21:16:30.266130 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.266025    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ecad8020-1da4-4c41-85ff-1edbf776141b-sys\") pod \"perf-node-gather-daemonset-vpqkz\" (UID: \"ecad8020-1da4-4c41-85ff-1edbf776141b\") " pod="openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz"
Sep 09 21:16:30.266130 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.266041    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ecad8020-1da4-4c41-85ff-1edbf776141b-podres\") pod \"perf-node-gather-daemonset-vpqkz\" (UID: \"ecad8020-1da4-4c41-85ff-1edbf776141b\") " pod="openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz"
Sep 09 21:16:30.266205 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.266156    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ecad8020-1da4-4c41-85ff-1edbf776141b-lib-modules\") pod \"perf-node-gather-daemonset-vpqkz\" (UID: \"ecad8020-1da4-4c41-85ff-1edbf776141b\") " pod="openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz"
Sep 09 21:16:30.266245 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.266213    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-692w5\" (UniqueName: \"kubernetes.io/projected/ecad8020-1da4-4c41-85ff-1edbf776141b-kube-api-access-692w5\") pod \"perf-node-gather-daemonset-vpqkz\" (UID: \"ecad8020-1da4-4c41-85ff-1edbf776141b\") " pod="openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz"
Sep 09 21:16:30.367059 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.367033    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ecad8020-1da4-4c41-85ff-1edbf776141b-sys\") pod \"perf-node-gather-daemonset-vpqkz\" (UID: \"ecad8020-1da4-4c41-85ff-1edbf776141b\") " pod="openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz"
Sep 09 21:16:30.367140 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.367062    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ecad8020-1da4-4c41-85ff-1edbf776141b-podres\") pod \"perf-node-gather-daemonset-vpqkz\" (UID: \"ecad8020-1da4-4c41-85ff-1edbf776141b\") " pod="openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz"
Sep 09 21:16:30.367140 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.367111    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ecad8020-1da4-4c41-85ff-1edbf776141b-lib-modules\") pod \"perf-node-gather-daemonset-vpqkz\" (UID: \"ecad8020-1da4-4c41-85ff-1edbf776141b\") " pod="openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz"
Sep 09 21:16:30.367239 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.367148    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-692w5\" (UniqueName: \"kubernetes.io/projected/ecad8020-1da4-4c41-85ff-1edbf776141b-kube-api-access-692w5\") pod \"perf-node-gather-daemonset-vpqkz\" (UID: \"ecad8020-1da4-4c41-85ff-1edbf776141b\") " pod="openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz"
Sep 09 21:16:30.367239 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.367193    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ecad8020-1da4-4c41-85ff-1edbf776141b-proc\") pod \"perf-node-gather-daemonset-vpqkz\" (UID: \"ecad8020-1da4-4c41-85ff-1edbf776141b\") " pod="openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz"
Sep 09 21:16:30.367239 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.367147    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ecad8020-1da4-4c41-85ff-1edbf776141b-sys\") pod \"perf-node-gather-daemonset-vpqkz\" (UID: \"ecad8020-1da4-4c41-85ff-1edbf776141b\") " pod="openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz"
Sep 09 21:16:30.367239 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.367227    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ecad8020-1da4-4c41-85ff-1edbf776141b-podres\") pod \"perf-node-gather-daemonset-vpqkz\" (UID: \"ecad8020-1da4-4c41-85ff-1edbf776141b\") " pod="openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz"
Sep 09 21:16:30.367413 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.367257    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ecad8020-1da4-4c41-85ff-1edbf776141b-lib-modules\") pod \"perf-node-gather-daemonset-vpqkz\" (UID: \"ecad8020-1da4-4c41-85ff-1edbf776141b\") " pod="openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz"
Sep 09 21:16:30.367413 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.367266    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ecad8020-1da4-4c41-85ff-1edbf776141b-proc\") pod \"perf-node-gather-daemonset-vpqkz\" (UID: \"ecad8020-1da4-4c41-85ff-1edbf776141b\") " pod="openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz"
Sep 09 21:16:30.375341 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.375323    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-692w5\" (UniqueName: \"kubernetes.io/projected/ecad8020-1da4-4c41-85ff-1edbf776141b-kube-api-access-692w5\") pod \"perf-node-gather-daemonset-vpqkz\" (UID: \"ecad8020-1da4-4c41-85ff-1edbf776141b\") " pod="openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz"
Sep 09 21:16:30.431324 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.431305    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz"
Sep 09 21:16:30.701355 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.701330    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz"]
Sep 09 21:16:30.703239 ip-10-0-135-117 kubenswrapper[2556]: W0909 21:16:30.703216    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podecad8020_1da4_4c41_85ff_1edbf776141b.slice/crio-7c25c70488054da8c6fb41be8d7a86b72aa10fd6754caaf88f0d3031ae88e84a WatchSource:0}: Error finding container 7c25c70488054da8c6fb41be8d7a86b72aa10fd6754caaf88f0d3031ae88e84a: Status 404 returned error can't find the container with id 7c25c70488054da8c6fb41be8d7a86b72aa10fd6754caaf88f0d3031ae88e84a
Sep 09 21:16:30.781949 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.781921    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz" event={"ID":"ecad8020-1da4-4c41-85ff-1edbf776141b","Type":"ContainerStarted","Data":"7dec389242409b3c57e17dfa7917939e07f99cb6f07e09a8b69fb3c11eae22a5"}
Sep 09 21:16:30.782060 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.781986    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz"
Sep 09 21:16:30.782060 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.782012    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz" event={"ID":"ecad8020-1da4-4c41-85ff-1edbf776141b","Type":"ContainerStarted","Data":"7c25c70488054da8c6fb41be8d7a86b72aa10fd6754caaf88f0d3031ae88e84a"}
Sep 09 21:16:30.799740 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:30.799701    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz" podStartSLOduration=0.799689493 podStartE2EDuration="799.689493ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:16:30.797746854 +0000 UTC m=+2636.477707163" watchObservedRunningTime="2026-09-09 21:16:30.799689493 +0000 UTC m=+2636.479649801"
Sep 09 21:16:32.468731 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:32.468704    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xxt4f_53e51d8b-4e88-46d9-9b73-d5c9b354ec0b/dns/0.log"
Sep 09 21:16:32.492743 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:32.492716    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xxt4f_53e51d8b-4e88-46d9-9b73-d5c9b354ec0b/kube-rbac-proxy/0.log"
Sep 09 21:16:32.548293 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:32.548268    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bmnnt_61416974-f481-4e36-ab4a-2be050c06afe/dns-node-resolver/0.log"
Sep 09 21:16:33.108802 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:33.108769    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-676b97c989-qctgc_6915fa52-02e5-4c44-894d-cb7083b1fc99/registry/0.log"
Sep 09 21:16:33.161542 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:33.161519    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kj4jd_c31d1698-1b9c-451e-995a-ed666693b6a2/node-ca/0.log"
Sep 09 21:16:34.743128 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:34.743100    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65d657c8d5-ptlv5_b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d/kube-auth-proxy/0.log"
Sep 09 21:16:35.367085 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:35.367058    2556 scope.go:117] "RemoveContainer" containerID="816230f0a1c329d41fc7056ebce756b3a5c5ad2c9b616e5b72cfd7e2d98d52d5"
Sep 09 21:16:35.374863 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:35.374845    2556 scope.go:117] "RemoveContainer" containerID="46804a8857ab054f120296837ccc3a703485d464abf611f2f828bf72916685f2"
Sep 09 21:16:35.803454 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:35.803434    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vtzkg_420e8660-43a1-4d78-841b-75448e999ac8/serve-healthcheck-canary/0.log"
Sep 09 21:16:36.449720 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:36.449691    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nmmbf_53ffa66d-d8a1-445f-a1cf-14d11c651095/kube-rbac-proxy/0.log"
Sep 09 21:16:36.473035 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:36.473008    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nmmbf_53ffa66d-d8a1-445f-a1cf-14d11c651095/exporter/0.log"
Sep 09 21:16:36.499816 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:36.499794    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nmmbf_53ffa66d-d8a1-445f-a1cf-14d11c651095/extractor/0.log"
Sep 09 21:16:36.796601 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:36.796574    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8prvh/perf-node-gather-daemonset-vpqkz"
Sep 09 21:16:39.821917 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:39.821883    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-859cf57674-j278h_50d78a07-27e0-41e1-9ecf-a9b9882eda59/manager/0.log"
Sep 09 21:16:39.847645 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:39.847624    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-859cf57674-j278h_50d78a07-27e0-41e1-9ecf-a9b9882eda59/copy-manifests/0.log"
Sep 09 21:16:40.468342 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:40.468275    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65d657c8d5-ptlv5_b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d/kube-auth-proxy/0.log"
Sep 09 21:16:41.267224 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:41.267178    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6cdf7f9947-sk8cw_859c9262-88b7-4d3b-a392-167f260fbb8a/manager/0.log"
Sep 09 21:16:42.798544 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:42.798456    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65d657c8d5-ptlv5_b4b0a5e1-d4b8-4dcc-ac2a-57f1ccee756d/kube-auth-proxy/0.log"
Sep 09 21:16:47.746870 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:47.746839    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m86gm_21bf48f4-01d2-4e62-af48-52956be58f15/kube-multus-additional-cni-plugins/0.log"
Sep 09 21:16:47.768606 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:47.768582    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m86gm_21bf48f4-01d2-4e62-af48-52956be58f15/egress-router-binary-copy/0.log"
Sep 09 21:16:47.789358 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:47.789340    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m86gm_21bf48f4-01d2-4e62-af48-52956be58f15/cni-plugins/0.log"
Sep 09 21:16:47.809272 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:47.809249    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m86gm_21bf48f4-01d2-4e62-af48-52956be58f15/bond-cni-plugin/0.log"
Sep 09 21:16:47.829940 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:47.829899    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m86gm_21bf48f4-01d2-4e62-af48-52956be58f15/routeoverride-cni/0.log"
Sep 09 21:16:47.851265 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:47.851243    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m86gm_21bf48f4-01d2-4e62-af48-52956be58f15/whereabouts-cni-bincopy/0.log"
Sep 09 21:16:47.874415 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:47.874398    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m86gm_21bf48f4-01d2-4e62-af48-52956be58f15/whereabouts-cni/0.log"
Sep 09 21:16:48.307779 ip-10-0-135-117 kubenswrapper[2556]: I0909 21:16:48.307756    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wkmrq_8c1a2e29-fbc8-4b78-99d0-ae382f56d9f3/kube-multus/0.log"